Basics of Probability Theory (for Theory of Computation courses)
|
|
- Rosalyn Carter
- 5 years ago
- Views:
Transcription
1 Basics of Probability Theory (for Theory of Computatio courses) Oded Goldreich Departmet of Computer Sciece Weizma Istitute of Sciece Rehovot, Israel. November 24, 2008 Preface. Probability theory plays a cetral role i may areas of computer sciece, ad specifically i cryptography ad complexity theory. I this text, we preset the basic probabilistic otios ad otatios that are used i various courses i the theory of computatio. Specifically, we refer to the otios of discrete probability space ad radom variables, ad to correspodig otatios. Also icluded are overviews of three useful probabilistic iequalities: Markov s Iequality, Chebyshev s Iequality, ad Cheroff Boud. 1 The Very Basics Formally speakig, probability theory is merely a quatitative study of the relatio betwee the sizes of various sets. I the simple case, which uderlies our applicatios, these sets are fiite ad the probability of a evet is merely a shorthad for the desity of a correspodig set with respect to a uderlyig (fiite) uiverse. For example, whe we talk of the probability that a coi flip yields the outcome heads, the uiverse is the set of the two possible outcomes (i.e., heads ad tails) ad the evet we refer to is a subset of this uiverse (i.e., the sigleto set heads). I geeral, the uiverse correspods to all possible basic situatios (which are assumed or postulated to be equally likely), ad evets correspod to specific sets of some of these situatios. Thus, oe may claim that probability theory is just combiatorics; however, as is ofte the case, good otatios (let aloe otios) are istrumetal to more complex studies. Throughout the etire text we refer oly to discrete probability distributios. Such probability distributios refer to a fiite uiverse, called the probability space (or the sample space), ad to subsets of this uiverse. Specifically, the uderlyig probability (or sample) space cosists of a fiite set, deoted Ω, ad evets correspod to subsets of Ω. The probability of a evet A Ω is defied as A / Ω. Ideed, oe should thik of probabilities by referrig to a (metal) experimet i which a elemet i the space Ω is selected with uiform probability distributio (i.e., each elemet is selected with equal probability, 1/ Ω ). Radom variables. Traditioally, radom variables are defied as fuctios from the sample space to the reals. (For example, for ay A Ω, we may cosider a radom variable X : Ω R such that X(e) = 1 of e A ad X(e) = 0 otherwise.) The probability that X is assiged a 1
2 particular value v is deoted Pr[X = v ad is defied as the probability of the evet {e Ω : X(e)}; that is, Pr[X =v = {e Ω : X(e)=v} / Ω. The support of a radom variable is the (fiite) set of values that are assiged positive probability; that is, the support of X is the set {X(e) : e Ω}. The expectatio of a radom variable X, defied over Ω, is defied as the average value of X whe the uderlyig sample is selected uiformly i Ω (i.e., Ω 1 e Ω X(e)). We deote this expectatio by E[X. A key observatio, referred to as liearity of expectatio, is that for ay two radom variables X, Y : Ω R it holds that E[X + Y = E[X + E[Y. I particular, for ay costats a ad b, it holds that E[aX + b = ae[x + b. The variace of a radom variable X, deoted Var[X, is defied as E[(X E[X) 2. Note that µ def = E[X is a costat, ad thus Var[X = E[(X µ) 2 = E[X 2 2µ X + µ 2 = E[X 2 2µ E[X + µ 2 = E[X 2 E[X 2 which is upper bouded by E[X 2. Note that a radom variable has positive variace if ad oly if it is ot a costat (i.e., Var[X > 0 if ad oly if X assumes at least two differet values). It is commo practice to talk about radom variables without specifyig the uderlyig probability space. I these cases a radom varibale is viewed as a discrete probability distributio over the reals; that is, we fix the probability that this radom variable is assiged ay value. Assumig that this probabilities are all ratioales, a correspodig probability space always exists. For example, whe we cosider a radom variable X such that Pr[X = 1 = 1/3 ad Pr[X = 2 = 2/3, a possible correspodig probability space is {1, 2, 3} such that X(1) = 1 ad X(2) = X(3) = 2. Statistical differece. The statistical distace (a.k.a variatio distace) betwee the radom variables X ad Y is defied as 1 2 Pr[X = v Pr[Y = v = max{pr[x S Pr[Y S}. (1) S v We say that X is δ-close (resp., δ-far) to Y if the statistical distace betwee them is at most (resp., at least) δ. Our Notatioal Covetios Typically, i our courses, the uderlyig probability space will cosist of the set of all strigs of a certai legth l. That is, the sample space is the set of all l-bit log strigs, ad each such strig is assiged probability measure 2 l. Abusig the traditioal termiology, we use the term radom variable also whe referrig to fuctios mappig the sample space ito the set of biary strigs. We ofte do ot specify the probability space, but rather talk directly about radom variables. For example, we may say that X is a radom variable assiged values i the set of all strigs such that Pr[X = 00 = 1 4 ad Pr[X = 111 = 3 4. (Such a radom variable may be defied over the sample space {0, 1}2, so that X(11) = 00 ad X(00) = X(01) = X(10) = 111.) Oe importat case of a radom variable is the output of a radomized process (e.g., a probabilistic polyomial-time algorithm). All our probabilistic statemets refer to radom variables that are defied beforehad. Typically, we may write Pr[f(X) = 1, where X is a radom variable defied beforehad (ad f is a 2
3 fuctio). A importat covetio is that all occurreces of the same symbol i a probabilistic statemet refer to the same (uique) radom variable. Hece, if B(, ) is a Boolea expressio depedig o two variables, ad X is a radom variable the Pr[B(X, X) deotes the probability that B(x, x) holds whe x is chose with probability Pr[X = x. For example, for every radom variable X, we have Pr[X =X = 1. We stress that if we wish to discuss the probability that B(x, y) holds whe x ad y are chose idepedetly with idetical probability distributio, the we will defie two idepedet radom variables each with the same probability distributio. 1 Hece, if X ad Y are two idepedet radom variables, the Pr[B(X, Y ) deotes the probability that B(x, y) holds whe the pair (x, y) is chose with probability Pr[X =x Pr[Y =y. For example, for every two idepedet radom variables, X ad Y, we have Pr[X = Y = 1 oly if both X ad Y are trivial (i.e., assig the etire probability mass to a sigle strig). We will ofte use U to deote a radom variable uiformly distributed over the set of all strigs of legth. Namely, Pr[U = α equals 2 if α {0, 1} ad equals 0 otherwise. We ofte refer to the distributio of U as the uiform distributio (eglectig to qualify that it is uiform over {0, 1} ). I additio, we occasioally use radom variables (arbitrarily) distributed over {0, 1} or {0, 1} l(), for some fuctio l : N N. Such radom variables are typically deoted by X, Y, Z, etc. We stress that i some cases X is distributed over {0, 1}, whereas i other cases it is distributed over {0, 1} l(), for some fuctio l (which is typically a polyomial). We ofte talk about probability esembles, which are ifiite sequece of radom variables {X } N such that each X rages over strigs of legth bouded by a polyomial i. 2 Three Iequalities The followig probabilistic iequalities are very useful. These iequalities refer to radom variables that are assiged real values ad provide upper-bouds o the probability that the radom variable deviates from its expectatio. 2.1 Markov s Iequality The most basic iequality is Markov s Iequality that applies to ay radom variable with bouded maximum or miimum value. For simplicity, this iequality is stated for radom variables that are lower-bouded by zero, ad reads as follows: Let X be a o-egative radom variable ad v be a o-egative real umber. The Pr [X v E(X) (2) v Equivaletly, Pr[X r E(X) 1 r. The proof amouts to the followig sequece: E(X) = x Pr[X =x x Pr[X =x 0 + Pr[X =x v x<v x v = Pr[X v v 1 Two radom variables, X ad Y, are called idepedet if for every pair of possible values (x, y) it holds that Pr[(X, Y )=(x, y) = Pr[X =x Pr[Y =y. 3
4 2.2 Chebyshev s Iequality Usig Markov s iequality, oe gets a potetially stroger boud o the deviatio of a radom variable from its expectatio. This boud, called Chebyshev s iequality, is useful whe havig additioal iformatio cocerig the radom variable (specifically, a good upper boud o its variace). For a radom variable X of fiite expectatio, we deote by Var(X) def = E[(X E(X)) 2 the variace of X, ad observe that Var(X) = E(X 2 ) E(X) 2. Chebyshev s Iequality the reads as follows: Let X be a radom variable, ad δ > 0. The Pr [ X E(X) δ Var(X) δ 2. Proof: We defie a radom variable Y def = (X E(X)) 2, ad apply Markov s iequality. We get Pr [ X E(X) δ = Pr [(X E(X)) 2 δ 2 ad the claim follows. E[(X E(X))2 δ 2 (3) Corollary (Pairwise Idepedet Samplig): Chebyshev s iequality is particularly useful i the aalysis of the error probability of approximatio via repeated samplig. It suffices to assume that the samples are picked i a pairwise idepedet maer, where X 1, X 2,..., X are pairwise idepedet if for every i j ad every α, β it holds that Pr[X i =α X j =β = Pr[X i =α Pr[X j =β. The corollary reads as follows: Let X 1, X 2,..., X be pairwise idepedet radom variables with idetical expectatio, deoted µ, ad idetical variace, deoted σ 2. The, for every ɛ > 0, it holds that [ X i Pr µ ɛ σ2 ɛ 2 (4). def Proof: Defie the radom variables X i = X i E(X i ). Note that the X i s are pairwise idepedet, ad each has zero expectatio. Applyig Chebyshev s iequality to the radom variable X i, ad usig the liearity of the expectatio operator, we get [ X i Pr µ ɛ Var [ X i E ɛ 2 [ ( X i = ɛ 2 2 Now (agai usig the liearity of expectatio) ( ) 2 [ E X i = E X 2 i + E [X i X j 1 i j By the pairwise idepedece of the X i s, we get E[X i X j = E[X i E[X j, ad usig E[X i = 0, we get ( ) 2 E X i = σ 2 The corollary follows. ) 2 4
5 2.3 Cheroff Boud Whe usig pairwise idepedet sample poits, the error probability i the approximatio decreases liearly with the umber of sample poits (see Eq. (4)). Whe usig totally idepedet sample poits, the error probability i the approximatio ca be show to decrease expoetially with the umber of sample poits. (Recall that the radom variables X 1, X 2,..., X are said to be totally idepedet if for every sequece a 1, a 2,..., a it holds that Pr[ X i =a i = Pr[X i =a i.) Probability bouds supportig the foregoig statemet are give ext. The first boud, commoly referred to as Cheroff Boud, cocers 0-1 radom variables (i.e., radom variables that are assiged as values either 0 or 1), ad asserts the followig. Let p 1 2, ad X 1, X 2,..., X be idepedet 0-1 radom variables such that Pr[X i = 1 = p, for each i. The, for every ɛ (0, p, it holds that [ X i Pr p > ɛ < 2 e c ɛ2, where c = max(2, 1 3p ). (5) The more commo formulatio sets c = 2, but the case c = 1/3p is very useful whe p is small ad oe cares about a multiplicative deviatio (e.g., ɛ = p/2). Proof Sketch: We upper-boud Pr[ X i p > ɛ, ad Pr[p X i > ɛ is bouded def similarly. Lettig X i = X i E(X i ), we apply Markov s iequality to the radom variable e λ X i, where λ (0, 1 will be determied to optimize the expressios that we derive. Thus, Pr[ X i > ɛ is upper-bouded by E[e λ X i e λɛ = e λɛ E[e λx i where the equality is due to the idepedece of the radom variables. To simplify the rest of the proof, we establish a sub-optimal boud as follows. Usig a Taylor expasio of e x (e.g., e x < 1 + x + x 2 for x 1) ad observig that E[X i = 0, we get E[e λx i < 1 + λ 2 E[X 2 i, which equals 1 + λ 2 p(1 p). Thus, Pr[ X i p > ɛ is upper-bouded by e λɛ (1 + λ 2 p(1 p)) < exp( λɛ + λ 2 p(1 p)), which is optimized at λ = ɛ/(2p(1 p)) yieldig exp( ɛ2 4p(1 p) ) exp( ɛ 2 ). The foregoig proof strategy ca be applied i more geeral settigs. 2 A more geeral boud, which refers to idepedet radom variables that are each bouded but are ot ecessarily idetical, is give ext (ad is commoly referred to as Hoefdig Iequality). Let X 1, X 2,..., X be idepedet radom variables, each ragig i the (real) iterval [a, b, ad let µ def = 1 E(X i ) deote the average expected value of these variables. The, for every ɛ > 0, [ X i Pr µ > ɛ < 2 e 2ɛ 2 (b a) 2 (6) The special case (of Eq. (6)) that refers to idetically distributed radom variables is easy to derive from the foregoig Cheroff Boud (by recallig Footote 2 ad usig a liear mappig of the iterval [a, b to the iterval [0, 1). This special case is useful i estimatig the average value of a (bouded) fuctio defied over a large domai, especially whe the desired error probability eeds to be egligible (i.e., decrease faster tha ay polyomial i the umber of samples). Such a estimate ca be obtaied provided that we ca sample the fuctio s domai (ad evaluate the fuctio). 2 For example, verify that the curret proof actually applies to the case that X i [0, 1 rather tha X i {0, 1}, by otig that Var[X i p(1 p) still holds. 5
6 2.4 Pairwise idepedet versus totally idepedet samplig I Sectios 2.2 ad 2.3 we cosidered two Laws of Large Numbers that assert that, whe sufficietly may trials are made, the average value obtaied i these actual trials typically approaches the expected value of a trial. I Sectio 2.2 these trials were performed based o pairwise idepedet samples, whereas i Sectio 2.3 these trials were performed based o totally idepedet samples. I this sectio we shall see that the amout of deviatio (of the average from the expectatio) is approximately the same i both cases, but the probability of deviatio is much smaller i the latter case. To demostrate the differece betwee the samplig bouds provided i Sectios 2.2 ad 2.3, we cosider the problem of estimatig the average value of a fuctio f : Ω [0, 1. I geeral, we say that a radom variable Z provides a (ɛ, δ)-approximatio of a value v if Pr[ Z v > ɛ δ. By Eq. (6), the average value of f evaluated at = O((ɛ 2 log(1/δ)) idepedet samples (selected uiformly i Ω) yield a (ɛ, δ)-approximatio of µ = x Ω f(x)/ Ω. Thus, the umber of sample poits is polyomially related to ɛ 1 ad logarithmically related to δ 1. I cotrast, by Eq. (4), a (ɛ, δ)-approximatio by pairwise idepedet samples calls for settig = O(ɛ 2 δ 1 ). We stress that, i both cases the umber of samples is polyomially related to the desired accuracy of the estimatio (i.e., ɛ). The oly advatage of totally idepedet samples over pairwise idepedet oes is i the depedecy of the umber of samples o the error probability (i.e., δ). 6
An Introduction to Randomized Algorithms
A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis
More informationDiscrete Mathematics for CS Spring 2008 David Wagner Note 22
CS 70 Discrete Mathematics for CS Sprig 2008 David Wager Note 22 I.I.D. Radom Variables Estimatig the bias of a coi Questio: We wat to estimate the proportio p of Democrats i the US populatio, by takig
More informationProblem Set 2 Solutions
CS271 Radomess & Computatio, Sprig 2018 Problem Set 2 Solutios Poit totals are i the margi; the maximum total umber of poits was 52. 1. Probabilistic method for domiatig sets 6pts Pick a radom subset S
More informationAdvanced Stochastic Processes.
Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.
More informationThe standard deviation of the mean
Physics 6C Fall 20 The stadard deviatio of the mea These otes provide some clarificatio o the distictio betwee the stadard deviatio ad the stadard deviatio of the mea.. The sample mea ad variace Cosider
More informationDistribution of Random Samples & Limit theorems
STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to
More informationEE 4TM4: Digital Communications II Probability Theory
1 EE 4TM4: Digital Commuicatios II Probability Theory I. RANDOM VARIABLES A radom variable is a real-valued fuctio defied o the sample space. Example: Suppose that our experimet cosists of tossig two fair
More informationRandom Variables, Sampling and Estimation
Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig
More informationOn Random Line Segments in the Unit Square
O Radom Lie Segmets i the Uit Square Thomas A. Courtade Departmet of Electrical Egieerig Uiversity of Califoria Los Ageles, Califoria 90095 Email: tacourta@ee.ucla.edu I. INTRODUCTION Let Q = [0, 1] [0,
More informationThis section is optional.
4 Momet Geeratig Fuctios* This sectio is optioal. The momet geeratig fuctio g : R R of a radom variable X is defied as g(t) = E[e tx ]. Propositio 1. We have g () (0) = E[X ] for = 1, 2,... Proof. Therefore
More informationLecture 2: Concentration Bounds
CSE 52: Desig ad Aalysis of Algorithms I Sprig 206 Lecture 2: Cocetratio Bouds Lecturer: Shaya Oveis Ghara March 30th Scribe: Syuzaa Sargsya Disclaimer: These otes have ot bee subjected to the usual scrutiy
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 2 9/9/2013. Large Deviations for i.i.d. Random Variables
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 2 9/9/2013 Large Deviatios for i.i.d. Radom Variables Cotet. Cheroff boud usig expoetial momet geeratig fuctios. Properties of a momet
More informationStatisticians use the word population to refer the total number of (potential) observations under consideration
6 Samplig Distributios Statisticias use the word populatio to refer the total umber of (potetial) observatios uder cosideratio The populatio is just the set of all possible outcomes i our sample space
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationn outcome is (+1,+1, 1,..., 1). Let the r.v. X denote our position (relative to our starting point 0) after n moves. Thus X = X 1 + X 2 + +X n,
CS 70 Discrete Mathematics for CS Sprig 2008 David Wager Note 9 Variace Questio: At each time step, I flip a fair coi. If it comes up Heads, I walk oe step to the right; if it comes up Tails, I walk oe
More information1 Convergence in Probability and the Weak Law of Large Numbers
36-752 Advaced Probability Overview Sprig 2018 8. Covergece Cocepts: i Probability, i L p ad Almost Surely Istructor: Alessadro Rialdo Associated readig: Sec 2.4, 2.5, ad 4.11 of Ash ad Doléas-Dade; Sec
More information1 of 7 7/16/2009 6:06 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 6. Order Statistics Defiitios Suppose agai that we have a basic radom experimet, ad that X is a real-valued radom variable
More informationLecture 2. The Lovász Local Lemma
Staford Uiversity Sprig 208 Math 233A: No-costructive methods i combiatorics Istructor: Ja Vodrák Lecture date: Jauary 0, 208 Origial scribe: Apoorva Khare Lecture 2. The Lovász Local Lemma 2. Itroductio
More informationApproximations and more PMFs and PDFs
Approximatios ad more PMFs ad PDFs Saad Meimeh 1 Approximatio of biomial with Poisso Cosider the biomial distributio ( b(k,,p = p k (1 p k, k λ: k Assume that is large, ad p is small, but p λ at the limit.
More informationLearning Theory: Lecture Notes
Learig Theory: Lecture Notes Kamalika Chaudhuri October 4, 0 Cocetratio of Averages Cocetratio of measure is very useful i showig bouds o the errors of machie-learig algorithms. We will begi with a basic
More informationFall 2013 MTH431/531 Real analysis Section Notes
Fall 013 MTH431/531 Real aalysis Sectio 8.1-8. Notes Yi Su 013.11.1 1. Defiitio of uiform covergece. We look at a sequece of fuctios f (x) ad study the coverget property. Notice we have two parameters
More informationEECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1
EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum
More informationDiscrete Mathematics for CS Spring 2007 Luca Trevisan Lecture 22
CS 70 Discrete Mathematics for CS Sprig 2007 Luca Trevisa Lecture 22 Aother Importat Distributio The Geometric Distributio Questio: A biased coi with Heads probability p is tossed repeatedly util the first
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationf X (12) = Pr(X = 12) = Pr({(6, 6)}) = 1/36
Probability Distributios A Example With Dice If X is a radom variable o sample space S, the the probablity that X takes o the value c is Similarly, Pr(X = c) = Pr({s S X(s) = c} Pr(X c) = Pr({s S X(s)
More informationApplication to Random Graphs
A Applicatio to Radom Graphs Brachig processes have a umber of iterestig ad importat applicatios. We shall cosider oe of the most famous of them, the Erdős-Réyi radom graph theory. 1 Defiitio A.1. Let
More informationChapter 5. Inequalities. 5.1 The Markov and Chebyshev inequalities
Chapter 5 Iequalities 5.1 The Markov ad Chebyshev iequalities As you have probably see o today s frot page: every perso i the upper teth percetile ears at least 1 times more tha the average salary. I other
More informationExpectation and Variance of a random variable
Chapter 11 Expectatio ad Variace of a radom variable The aim of this lecture is to defie ad itroduce mathematical Expectatio ad variace of a fuctio of discrete & cotiuous radom variables ad the distributio
More information62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 +
62. Power series Defiitio 16. (Power series) Give a sequece {c }, the series c x = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + is called a power series i the variable x. The umbers c are called the coefficiets of
More informationFACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures
FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING Lectures MODULE 5 STATISTICS II. Mea ad stadard error of sample data. Biomial distributio. Normal distributio 4. Samplig 5. Cofidece itervals
More informationDiscrete Mathematics and Probability Theory Summer 2014 James Cook Note 15
CS 70 Discrete Mathematics ad Probability Theory Summer 2014 James Cook Note 15 Some Importat Distributios I this ote we will itroduce three importat probability distributios that are widely used to model
More informationLecture 6: Coupon Collector s problem
Radomized Algorithms Lecture 6: Coupo Collector s problem Sotiris Nikoletseas Professor CEID - ETY Course 2017-2018 Sotiris Nikoletseas, Professor Radomized Algorithms - Lecture 6 1 / 16 Variace: key features
More informationDiscrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 19
CS 70 Discrete Mathematics ad Probability Theory Sprig 2016 Rao ad Walrad Note 19 Some Importat Distributios Recall our basic probabilistic experimet of tossig a biased coi times. This is a very simple
More informationLecture 12: November 13, 2018
Mathematical Toolkit Autum 2018 Lecturer: Madhur Tulsiai Lecture 12: November 13, 2018 1 Radomized polyomial idetity testig We will use our kowledge of coditioal probability to prove the followig lemma,
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More information4. Partial Sums and the Central Limit Theorem
1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems
More informationLecture 9: Expanders Part 2, Extractors
Lecture 9: Expaders Part, Extractors Topics i Complexity Theory ad Pseudoradomess Sprig 013 Rutgers Uiversity Swastik Kopparty Scribes: Jaso Perry, Joh Kim I this lecture, we will discuss further the pseudoradomess
More informationRandomized Algorithms I, Spring 2018, Department of Computer Science, University of Helsinki Homework 1: Solutions (Discussed January 25, 2018)
Radomized Algorithms I, Sprig 08, Departmet of Computer Sciece, Uiversity of Helsiki Homework : Solutios Discussed Jauary 5, 08). Exercise.: Cosider the followig balls-ad-bi game. We start with oe black
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 6 9/24/2008 DISCRETE RANDOM VARIABLES AND THEIR EXPECTATIONS
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 6 9/24/2008 DISCRETE RANDOM VARIABLES AND THEIR EXPECTATIONS Cotets 1. A few useful discrete radom variables 2. Joit, margial, ad
More informationMathematical Statistics - MS
Paper Specific Istructios. The examiatio is of hours duratio. There are a total of 60 questios carryig 00 marks. The etire paper is divided ito three sectios, A, B ad C. All sectios are compulsory. Questios
More informationProbability 2 - Notes 10. Lemma. If X is a random variable and g(x) 0 for all x in the support of f X, then P(g(X) 1) E[g(X)].
Probability 2 - Notes 0 Some Useful Iequalities. Lemma. If X is a radom variable ad g(x 0 for all x i the support of f X, the P(g(X E[g(X]. Proof. (cotiuous case P(g(X Corollaries x:g(x f X (xdx x:g(x
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationUnbiased Estimation. February 7-12, 2008
Ubiased Estimatio February 7-2, 2008 We begi with a sample X = (X,..., X ) of radom variables chose accordig to oe of a family of probabilities P θ where θ is elemet from the parameter space Θ. For radom
More informationDisjoint Systems. Abstract
Disjoit Systems Noga Alo ad Bey Sudaov Departmet of Mathematics Raymod ad Beverly Sacler Faculty of Exact Scieces Tel Aviv Uiversity, Tel Aviv, Israel Abstract A disjoit system of type (,,, ) is a collectio
More informationHypothesis Testing. Evaluation of Performance of Learned h. Issues. Trade-off Between Bias and Variance
Hypothesis Testig Empirically evaluatig accuracy of hypotheses: importat activity i ML. Three questios: Give observed accuracy over a sample set, how well does this estimate apply over additioal samples?
More informationLecture 01: the Central Limit Theorem. 1 Central Limit Theorem for i.i.d. random variables
CSCI-B609: A Theorist s Toolkit, Fall 06 Aug 3 Lecture 0: the Cetral Limit Theorem Lecturer: Yua Zhou Scribe: Yua Xie & Yua Zhou Cetral Limit Theorem for iid radom variables Let us say that we wat to aalyze
More informationStochastic Simulation
Stochastic Simulatio 1 Itroductio Readig Assigmet: Read Chapter 1 of text. We shall itroduce may of the key issues to be discussed i this course via a couple of model problems. Model Problem 1 (Jackso
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationf X (12) = Pr(X = 12) = Pr({(6, 6)}) = 1/36
Probability Distributios A Example With Dice If X is a radom variable o sample space S, the the probability that X takes o the value c is Similarly, Pr(X = c) = Pr({s S X(s) = c}) Pr(X c) = Pr({s S X(s)
More informationAxioms of Measure Theory
MATH 532 Axioms of Measure Theory Dr. Neal, WKU I. The Space Throughout the course, we shall let X deote a geeric o-empty set. I geeral, we shall ot assume that ay algebraic structure exists o X so that
More informationLecture 19: Convergence
Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may
More information( ) = p and P( i = b) = q.
MATH 540 Radom Walks Part 1 A radom walk X is special stochastic process that measures the height (or value) of a particle that radomly moves upward or dowward certai fixed amouts o each uit icremet of
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationCS166 Handout 02 Spring 2018 April 3, 2018 Mathematical Terms and Identities
CS166 Hadout 02 Sprig 2018 April 3, 2018 Mathematical Terms ad Idetities Thaks to Ady Nguye ad Julie Tibshirai for their advice o this hadout. This hadout covers mathematical otatio ad idetities that may
More informationRandom Walks on Discrete and Continuous Circles. by Jeffrey S. Rosenthal School of Mathematics, University of Minnesota, Minneapolis, MN, U.S.A.
Radom Walks o Discrete ad Cotiuous Circles by Jeffrey S. Rosethal School of Mathematics, Uiversity of Miesota, Mieapolis, MN, U.S.A. 55455 (Appeared i Joural of Applied Probability 30 (1993), 780 789.)
More informationThe Maximum-Likelihood Decoding Performance of Error-Correcting Codes
The Maximum-Lielihood Decodig Performace of Error-Correctig Codes Hery D. Pfister ECE Departmet Texas A&M Uiversity August 27th, 2007 (rev. 0) November 2st, 203 (rev. ) Performace of Codes. Notatio X,
More information5. INEQUALITIES, LIMIT THEOREMS AND GEOMETRIC PROBABILITY
IA Probability Let Term 5 INEQUALITIES, LIMIT THEOREMS AND GEOMETRIC PROBABILITY 51 Iequalities Suppose that X 0 is a radom variable takig o-egative values ad that c > 0 is a costat The P X c E X, c is
More informationMATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4
MATH 30: Probability ad Statistics 9. Estimatio ad Testig of Parameters Estimatio ad Testig of Parameters We have bee dealig situatios i which we have full kowledge of the distributio of a radom variable.
More informationInformation Theory and Statistics Lecture 4: Lempel-Ziv code
Iformatio Theory ad Statistics Lecture 4: Lempel-Ziv code Łukasz Dębowski ldebowsk@ipipa.waw.pl Ph. D. Programme 203/204 Etropy rate is the limitig compressio rate Theorem For a statioary process (X i)
More informationAda Boost, Risk Bounds, Concentration Inequalities. 1 AdaBoost and Estimates of Conditional Probabilities
CS8B/Stat4B Sprig 008) Statistical Learig Theory Lecture: Ada Boost, Risk Bouds, Cocetratio Iequalities Lecturer: Peter Bartlett Scribe: Subhrasu Maji AdaBoost ad Estimates of Coditioal Probabilities We
More informationIntroduction to Probability. Ariel Yadin
Itroductio to robability Ariel Yadi Lecture 2 *** Ja. 7 ***. Covergece of Radom Variables As i the case of sequeces of umbers, we would like to talk about covergece of radom variables. There are may ways
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More informationECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors
ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic
More informationSimulation. Two Rule For Inverting A Distribution Function
Simulatio Two Rule For Ivertig A Distributio Fuctio Rule 1. If F(x) = u is costat o a iterval [x 1, x 2 ), the the uiform value u is mapped oto x 2 through the iversio process. Rule 2. If there is a jump
More information1 Introduction to reducing variance in Monte Carlo simulations
Copyright c 010 by Karl Sigma 1 Itroductio to reducig variace i Mote Carlo simulatios 11 Review of cofidece itervals for estimatig a mea I statistics, we estimate a ukow mea µ = E(X) of a distributio by
More informationIf a subset E of R contains no open interval, is it of zero measure? For instance, is the set of irrationals in [0, 1] is of measure zero?
2 Lebesgue Measure I Chapter 1 we defied the cocept of a set of measure zero, ad we have observed that every coutable set is of measure zero. Here are some atural questios: If a subset E of R cotais a
More informationCS284A: Representations and Algorithms in Molecular Biology
CS284A: Represetatios ad Algorithms i Molecular Biology Scribe Notes o Lectures 3 & 4: Motif Discovery via Eumeratio & Motif Represetatio Usig Positio Weight Matrix Joshua Gervi Based o presetatios by
More informationMath 216A Notes, Week 5
Math 6A Notes, Week 5 Scribe: Ayastassia Sebolt Disclaimer: These otes are ot early as polished (ad quite possibly ot early as correct) as a published paper. Please use them at your ow risk.. Thresholds
More informationHOMEWORK 2 SOLUTIONS
HOMEWORK SOLUTIONS CSE 55 RANDOMIZED AND APPROXIMATION ALGORITHMS 1. Questio 1. a) The larger the value of k is, the smaller the expected umber of days util we get all the coupos we eed. I fact if = k
More informationDiscrete Mathematics and Probability Theory Fall 2016 Walrand Probability: An Overview
CS 70 Discrete Mathematics ad Probability Theory Fall 2016 Walrad Probability: A Overview Probability is a fasciatig theory. It provides a precise, clea, ad useful model of ucertaity. The successes of
More informationNotes 5 : More on the a.s. convergence of sums
Notes 5 : More o the a.s. covergece of sums Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: Dur0, Sectios.5; Wil9, Sectio 4.7, Shi96, Sectio IV.4, Dur0, Sectio.. Radom series. Three-series
More informationIt is always the case that unions, intersections, complements, and set differences are preserved by the inverse image of a function.
MATH 532 Measurable Fuctios Dr. Neal, WKU Throughout, let ( X, F, µ) be a measure space ad let (!, F, P ) deote the special case of a probability space. We shall ow begi to study real-valued fuctios defied
More informationSTAT Homework 1 - Solutions
STAT-36700 Homework 1 - Solutios Fall 018 September 11, 018 This cotais solutios for Homework 1. Please ote that we have icluded several additioal commets ad approaches to the problems to give you better
More informationDiscrete Mathematics and Probability Theory Spring 2012 Alistair Sinclair Note 15
CS 70 Discrete Mathematics ad Probability Theory Sprig 2012 Alistair Siclair Note 15 Some Importat Distributios The first importat distributio we leared about i the last Lecture Note is the biomial distributio
More informationLecture 4: April 10, 2013
TTIC/CMSC 1150 Mathematical Toolkit Sprig 01 Madhur Tulsiai Lecture 4: April 10, 01 Scribe: Haris Agelidakis 1 Chebyshev s Iequality recap I the previous lecture, we used Chebyshev s iequality to get a
More informationRademacher Complexity
EECS 598: Statistical Learig Theory, Witer 204 Topic 0 Rademacher Complexity Lecturer: Clayto Scott Scribe: Ya Deg, Kevi Moo Disclaimer: These otes have ot bee subjected to the usual scrutiy reserved for
More informationThis exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam.
Probability ad Statistics FS 07 Secod Sessio Exam 09.0.08 Time Limit: 80 Miutes Name: Studet ID: This exam cotais 9 pages (icludig this cover page) ad 0 questios. A Formulae sheet is provided with the
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS
MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak
More informationTopic 8: Expected Values
Topic 8: Jue 6, 20 The simplest summary of quatitative data is the sample mea. Give a radom variable, the correspodig cocept is called the distributioal mea, the epectatio or the epected value. We begi
More informationNOTES ON DISTRIBUTIONS
NOTES ON DISTRIBUTIONS MICHAEL N KATEHAKIS Radom Variables Radom variables represet outcomes from radom pheomea They are specified by two objects The rage R of possible values ad the frequecy fx with which
More informationEcon 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1.
Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chi-square Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio
More informationProbability and Random Processes
Probability ad Radom Processes Lecture 5 Probability ad radom variables The law of large umbers Mikael Skoglud, Probability ad radom processes 1/21 Why Measure Theoretic Probability? Stroger limit theorems
More information6 Integers Modulo n. integer k can be written as k = qn + r, with q,r, 0 r b. So any integer.
6 Itegers Modulo I Example 2.3(e), we have defied the cogruece of two itegers a,b with respect to a modulus. Let us recall that a b (mod ) meas a b. We have proved that cogruece is a equivalece relatio
More informationLecture 12: September 27
36-705: Itermediate Statistics Fall 207 Lecturer: Siva Balakrisha Lecture 2: September 27 Today we will discuss sufficiecy i more detail ad the begi to discuss some geeral strategies for costructig estimators.
More informationECE 901 Lecture 12: Complexity Regularization and the Squared Loss
ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality
More informationThe Boolean Ring of Intervals
MATH 532 Lebesgue Measure Dr. Neal, WKU We ow shall apply the results obtaied about outer measure to the legth measure o the real lie. Throughout, our space X will be the set of real umbers R. Whe ecessary,
More informationDiscrete Mathematics for CS Spring 2005 Clancy/Wagner Notes 21. Some Important Distributions
CS 70 Discrete Mathematics for CS Sprig 2005 Clacy/Wager Notes 21 Some Importat Distributios Questio: A biased coi with Heads probability p is tossed repeatedly util the first Head appears. What is the
More informationOutput Analysis and Run-Length Control
IEOR E4703: Mote Carlo Simulatio Columbia Uiversity c 2017 by Marti Haugh Output Aalysis ad Ru-Legth Cotrol I these otes we describe how the Cetral Limit Theorem ca be used to costruct approximate (1 α%
More informationLecture 2: April 3, 2013
TTIC/CMSC 350 Mathematical Toolkit Sprig 203 Madhur Tulsiai Lecture 2: April 3, 203 Scribe: Shubhedu Trivedi Coi tosses cotiued We retur to the coi tossig example from the last lecture agai: Example. Give,
More informationEmpirical Process Theory and Oracle Inequalities
Stat 928: Statistical Learig Theory Lecture: 10 Empirical Process Theory ad Oracle Iequalities Istructor: Sham Kakade 1 Risk vs Risk See Lecture 0 for a discussio o termiology. 2 The Uio Boud / Boferoi
More informationNANYANG TECHNOLOGICAL UNIVERSITY SYLLABUS FOR ENTRANCE EXAMINATION FOR INTERNATIONAL STUDENTS AO-LEVEL MATHEMATICS
NANYANG TECHNOLOGICAL UNIVERSITY SYLLABUS FOR ENTRANCE EXAMINATION FOR INTERNATIONAL STUDENTS AO-LEVEL MATHEMATICS STRUCTURE OF EXAMINATION PAPER. There will be oe 2-hour paper cosistig of 4 questios.
More informationLecture 2: Monte Carlo Simulation
STAT/Q SCI 43: Itroductio to Resamplig ethods Sprig 27 Istructor: Ye-Chi Che Lecture 2: ote Carlo Simulatio 2 ote Carlo Itegratio Assume we wat to evaluate the followig itegratio: e x3 dx What ca we do?
More informationMassachusetts Institute of Technology
Solutios to Quiz : Sprig 006 Problem : Each of the followig statemets is either True or False. There will be o partial credit give for the True False questios, thus ay explaatios will ot be graded. Please
More informationLecture 3: August 31
36-705: Itermediate Statistics Fall 018 Lecturer: Siva Balakrisha Lecture 3: August 31 This lecture will be mostly a summary of other useful expoetial tail bouds We will ot prove ay of these i lecture,
More informationMath 525: Lecture 5. January 18, 2018
Math 525: Lecture 5 Jauary 18, 2018 1 Series (review) Defiitio 1.1. A sequece (a ) R coverges to a poit L R (writte a L or lim a = L) if for each ǫ > 0, we ca fid N such that a L < ǫ for all N. If the
More informationProperties and Hypothesis Testing
Chapter 3 Properties ad Hypothesis Testig 3.1 Types of data The regressio techiques developed i previous chapters ca be applied to three differet kids of data. 1. Cross-sectioal data. 2. Time series data.
More informationON POINTWISE BINOMIAL APPROXIMATION
Iteratioal Joural of Pure ad Applied Mathematics Volume 71 No. 1 2011, 57-66 ON POINTWISE BINOMIAL APPROXIMATION BY w-functions K. Teerapabolar 1, P. Wogkasem 2 Departmet of Mathematics Faculty of Sciece
More informationInfinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
More informationSDS 321: Introduction to Probability and Statistics
SDS 321: Itroductio to Probability ad Statistics Lecture 23: Cotiuous radom variables- Iequalities, CLT Puramrita Sarkar Departmet of Statistics ad Data Sciece The Uiversity of Texas at Austi www.cs.cmu.edu/
More information