Entropy and Ergodic Theory Lecture 5: Joint typicality and conditional AEP
|
|
- Ashlee King
- 5 years ago
- Views:
Transcription
1 Etropy ad Ergodic Theory Lecture 5: Joit typicality ad coditioal AEP 1 Notatio: from RVs back to distributios Let (Ω, F, P) be a probability space, ad let X ad Y be A- ad B-valued discrete RVs, respectively. The p X,Y Prob(A B), ad we may write it usig the multiplicatio rule for coditioal probabilities: p X,Y (a, b) = p X (a) p Y X (b a). I vaious applicatios of the theory, we eed to fix oe of the two factors o the right of this equatio, ad cosider all the possible joit distributios that ca be obtaied by varyig the other factor. Defiitio 1.1. A kerel from A to B is a A B matrix θ = (θ(b a)) a,b such that the row vector (θ(b a)) b B is a probability vector o B for each a A. Later i this course we exted this defiitio to more geeral measurable spaces A ad B. The th extesio of a kerel θ is the kerel θ from A to B defied by θ (y 1 y x 1 x ) = θ(y i x i ). Each of the fuctios θ ( x) for x A is a product probability distributio o B. A kerel θ may be see as specifyig a collectio of coditioal probabilities θ( a), a A, but ot a probability distributio o A itself. If (X, Y ) are as above, the we say that p Y X agrees with a kerel θ if p Y X (b a) = θ(b a) wheever p X (a) > 0. 1
2 If p Prob(A) ad θ is a kerel from A to B, the they ca be combied to give a probability distributio o A B: q(a, b) := p(a) θ(b a). (1) This is sometimes called the iput-output distributio of p ad θ, or the hookup of p with θ. We sometimes deote if by the (o-stadard) otatio p θ, which is meat to suggest a kid of geeralizatio of a product measure. If (X, Y ) are a (A B)-valued RV with this distributio, the p X = p ad p Y X agrees with θ. Ay pair of RVs (X, Y ) with this distributio is called a radomizatio of p ad θ. If (X, Y ) is a radomizatio of p ad θ, the the coditioal etropy H(Y X) clearly depeds oly o the joit distributio p θ. We sometimes use the alterative otatio H(θ p) := H(Y X) = a A p(a) H(θ( a)). Sice this equals H(X, Y ) H(X) = H(p θ) H(p), it is clearly a cotiuous fuctio of the joit distributio p θ. 2 Joit typicality ad the coditioal AEP Let q Prob(A B), ad let p A ad p B its margials o A ad B. I order to emphasize that q is a distributio o a product of two alphabets, we sometimes refer to elemets of T,δ (q) as joitly δ-typical for q. This omeaclature also emphasizes the followig importat poit: If x T,δ (p A ) ad y T,δ (p B ), it does ot follow that (x, y) is joitly ε-typical for q for some small ε. Example. Let x {0, 1} be δ-typical for the uiform distributio o {0, 1}, ad let y = x. The (x, y) is very far from typical for the uiform distributio o {0, 1} {0, 1}, sice N(01 (x, y)) = N(10 (x, y)) = 0 /4. Istead, (x, y) is typical for the distributio p = 1 2 (δ 00 +δ 11 ) o {0, 1} {0, 1}. 2
3 Our ext goal is a coutig-problem itepretatio of coditioal etropy, aalogous to the meaig established for ucoditioal etropy i Lecture 1. I provig it, we will also meet coditioal versios of the LLN for types ad the AEP. We build up all these results followig roughly the same sequece as i the ucoditioal settig of Lecture 1. Let us ow write q = p A θ for some kerel θ from A to B. We fix all of these otatios for the rest of the lecture. 2.1 Coditioal typicality Defiitio 2.1. Let x A ad y B. The the coditioal type of y give x is the collectio of values p y x (b a) := N( (a, b) (x, y) ), N(a x) defied for all a A such that N(a x) > 0. Heceforth, we usually abbreviate N(ab xy) := N ( (a, b) (x, y) ). Ituitively, the quatity p y x (b a) does the followig: amog all i {1, 2,..., }, we cosider oly those for which x i = a, ad amog those we record the fractio which also satisfy y i = b. Sometimes we refer to the coditioal type as beig idexed by all a A, without worryig about whether N(a x) > 0. We do this oly i cases where it causes o real problems. The first part of our iterpretatio of coditioal etropy is the followig. It is the aalog of the upper boud i Theorem 3.1 from Lecture 1. Lemma 2.2. Fix a kerel θ ad a strig x A. The umber of strigs y B for which p y x agrees with θ is at most 2 H(θ px). Proof. Exercise: mimic the proof of the upper boud i Lecture 1, Theorem 3.1. The key fact is that, if p y x agrees with θ, the N(ab xy) = θ(b a)p x (a) for a A, b B. 3
4 Like Theorem 3.1 i Lecture 1, there is a matchig lower boud for this lemma, but we leave it aside here. Istead let us itroduce a approximate form of the problem, ad prove upper ad lower bouds for that. The approximate form is more importat for later applicatios. Defiitio 2.3. Let δ > 0 ad x A. The y B is coditioally δ-typical for θ give x if px,y p x θ < δ. The set of all such y is deoted by T,δ (θ, x). More expasively, y T,δ (θ, x) if N(ab xy) a,b N(a x) θ(b a) < δ. Beware that some authors use slightly differet defiitios, as i the case of ucoditioal tylicality. This defiitio requires just a little care for the followig reaso: if θ ad λ are two kerels satisfyig θ(b a) = λ(b a) wheever N(a x) > 0, the y is coditioally δ-typical for θ give x if ad oly if the same holds for λ. As i the ucoditioal settig, the upper boud of Lemma 2.2 is easily tured ito a upper boud for the problem of coutig approximately coditioally typical strigs. Corollary 2.4. For x A ad δ > 0 we have T,δ (θ, x) 2 H(θ px)+ (δ)+o(), where the estimates i the two error terms of the expoet are idepedet of x. Proof. The quatity H(θ p) is cotiuous as a fuctio of the joit distributio p θ. From this ad Lemma 2.2 it follows that {y : p y x agrees with λ} 2 H(λ px) H(θ px)+ (δ) 2 wheever p x λ p x θ < δ. The values p y x (b a) take by the coditioal type of y give x are all ratios of itegers betwee 0 ad, ad there are at most A B of these values. Therefore 4
5 we may choose a set C of at most ( + 1) 2 A B ratioal stochastic matrices such that every coditioal type betwee strigs of legth agrees with some λ C. Havig doe so, we obtai T,δ (θ, x) λ C p x λ p x θ < δ λ C p x λ p x θ < δ H(λ px) 2 2 H(θ px)+ (δ) 2 H(θ px)+ (δ)+o(). I the remaider of this sectio, we itroduce a LLN ad AEP for coditioal types, ad obtai a lower boud which complemets the previous corollary. 2.2 A LLN for coditioal types We eed a versio of the weak law of large umbers for idepedet RVs of possibly differet distributios. We give a very simple versio which assumes a uiform boud o the RVs. It ca be proved by the same applicatio of Chebyshev s iequality as i the i.i.d. case (which i fact requires oly a uiform boud o the variaces). Propositio 2.5 (Weak LLN for RVs with differig distributios). Fix M > 0. For ay ε > 0 there is a 0 N, depedig o M ad ε, with the followig property: wheever 0 ad X 1,... X are idepedet [ M, M]-valued RVs, we have { 1 P [ 1 X i E ] } X i > ε < ε. Propositio 2.6 (LLN for coditioal types). For ay δ > 0, we have as. mi θ( T,δ (θ, x) ) x 1 x A 5
6 Proof. Fix x A, ad let Y B be draw from the distributio θ ( x). This meas that Y 1,..., Y are idepedet ad that Y i has distributio θ( x i ). For ay (a, b) A B we have p x,y (a, b) = N(ab xy) = 1 1 {xi =a, Y i =b}. The RVs 1 {xi =a, Y i =b} are idepedet, but i geeral ot idetically distributed: for those i which satisfy x i = a the distributio is Beroulli(θ(b a)), ad for the other i this RV is idetically zero. However, it does follow that they are all bouded by 1, ad so Propositio 2.5 gives { px,y P (a, b) E [ p x,y (a, b) ] } < δ 1 δ > 0, where the rate of covergece does ot deped o the specific choice of x. This gives the correct limitig behaviour for p x,y (a, b) because E [ p x,y (a, b) ] = 1 i: x i =a E[1 {Yi =b}] = 1 N(a x)θ(b a) = (p x θ)(a, b). Combiig this covergece for the fiitely may choices of (a, b) completes the proof. 2.3 Coditioal etropy typicality ad AEP There is also a coditioal versio of etropy typicality. Defiitio 2.7. A strig y B is coditioally ε-etropy typical for θ give x if 2 H(θ px) ε < θ (y x) < 2 H(θ px)+ε. The set of all such y is deoted by T et,ε (θ, x). The choice of the costat H(θ p x ) here is justified by the followig. Theorem 2.8 (Coditioal AEP). For ay ε > 0 we have as. mi θ( T et x A,ε (θ, x) ) x 1 6
7 Proof. Let Y θ ( x) as before. Takig logarithms gives log 2 θ (Y x) = Z i, where Z i = log 2 θ(y i x i ). These are idepedet, fiite-valued RVs. Amog them, there are at most A -may differet possible distributios, oe for each of the possible values of x i. This list of possible distributios depeds o θ, but the specific choice of x determies oly which distributio is chose for each Z i. Therefore Propositio 2.5 gives { 1 [ 1 } P log 2 θ (Y x) E log 2 θ (Y x)] < δ 1 δ > 0, where the rate of covergece agai does ot deped o the specific choice of x. This completes the proof, because E[log 2 θ (Y x)] = = m E[log 2 θ(y i x i )] = H(θ( x i )) = a θ(b x i ) log 2 θ(b x i ) b p x (a) H(θ( a)) = H(θ P x ). From this we obtai a coverig-umber corollary exactly as i the ucoditioal settig. Corollary 2.9. For ay ε > 0, we have cov ε (θ ( x)) = 2 H(θ px)+oε(), where the estimate i o ε () also depeds o θ but ot o x. This ow implies the lower boud o the umber of coditioally typical strigs which accompaies Corollary 2.4 Corollary For ay x A we have T,δ (θ, x) 2 H(θ px) o δ(), where the estimate i o δ () depeds o θ but ot o x. Proof. Combie Propositio 2.6 with the previous corollary. 7
8 2.4 Ituitive picture of the above results Let (X, Y ) be a radomizatio of q. The origial AEP lets us roughly visualize q as a uiform distributio o typical strigs. If we wish to thik about q as a joit distributio for pairs of strigs (x, y), the the results of this sectio give us a ehaced versio of that picture. To explai this, we start with the followig. The followig lemma gives some simple but useful relatios betwee coditioal ad ucoditioal typicality. Lemma If x T,δ (p) ad y T,δ (θ, x) the (x, y) T,2δ (p θ). 2. O the other had, if (x, y) T,δ (p θ), the y T,2δ (θ, x). Proof. The key is that for ay p Prob(A) we have p θ p θ = p (a)θ(b a) p(a)θ(b a) a,b = a ( ) p (a) p(a) θ(b a) = p p. b Part 1. Usig the calculatio above, our first pair of assumptios gives p x,y p θ p x,y p x θ + p x θ p θ = p x,y p x θ + p x p < 2δ. Part 2. Clearly if (x, y) T,δ (p θ), the x T,δ (p). Give this, we obtai p x,y p x θ p x,y p θ + p x θ p θ < 2δ. More succictly, this lemma asserts that T,δ/2 (p θ) ( {x} T,δ (θ, x) ) T,2δ (p θ) δ > 0. (2) x T,δ (p) Usig (2), we see that q is somethig like the uiform distributio o the subset of the Cartesia product where moreover T,δ (q) T,δ (p A ) T,δ (p B ), 8
9 1. most vertical slices T,δ (q) ({x} B ) have size roughly 2 H(Y X) (where we restrict attetio to x which is itself typical for p A ), ad similarly 2. most horizotal slices T,δ (q) (A {y}) have size roughly 2 H(X Y ). This ituitive picture gives us a ice way to visualize the chai rule. There are some messy εs ad δs to keep track of, but the idea is this: 2 H(X,Y ) T,δ (q) = T,δ (q) ({x} B ) x T,δ (p A ) T,δ (p A ) (typical size T,δ (q) ({x} B ) ) 2 H(X) 2 H(Y X), where the approximatio betwee the first ad the secod lies uses property 1 above. 3 Notes ad remarks Basic sources for this lecture: [CT91, Chapter 10, Exercise 16]. Further readig: the book [CK11] icludes a much more thorough accout of typicality ad coditioal typicality. Refereces [CK11] Imre Csiszár ad Jáos Körer. Iformatio theory. Cambridge Uiversity Press, Cambridge, secod editio, Codig theorems for discrete memoryless systems. [CT91] Thomas M. Cover ad Joy A. Thomas. Elemets of iformatio theory. Wiley Series i Telecommuicatios. Joh Wiley & Sos, Ic., New York, A Wiley-Itersciece Publicatio. TIM AUSTIN COURANT INSTITUTE OF MATHEMATICAL SCIENCES, NEW YORK UNIVERSITY, 251 MERCER ST, NEW YORK NY 10012, USA tim@cims.yu.edu URL: cims.yu.edu/ tim 9
Lecture 6: Source coding, Typicality, and Noisy channels and capacity
15-859: Iformatio Theory ad Applicatios i TCS CMU: Sprig 2013 Lecture 6: Source codig, Typicality, ad Noisy chaels ad capacity Jauary 31, 2013 Lecturer: Mahdi Cheraghchi Scribe: Togbo Huag 1 Recap Uiversal
More informationProduct measures, Tonelli s and Fubini s theorems For use in MAT3400/4400, autumn 2014 Nadia S. Larsen. Version of 13 October 2014.
Product measures, Toelli s ad Fubii s theorems For use i MAT3400/4400, autum 2014 Nadia S. Larse Versio of 13 October 2014. 1. Costructio of the product measure The purpose of these otes is to preset the
More informationLecture 15: Strong, Conditional, & Joint Typicality
EE376A/STATS376A Iformatio Theory Lecture 15-02/27/2018 Lecture 15: Strog, Coditioal, & Joit Typicality Lecturer: Tsachy Weissma Scribe: Nimit Sohoi, William McCloskey, Halwest Mohammad I this lecture,
More informationLecture 14: Graph Entropy
15-859: Iformatio Theory ad Applicatios i TCS Sprig 2013 Lecture 14: Graph Etropy March 19, 2013 Lecturer: Mahdi Cheraghchi Scribe: Euiwoog Lee 1 Recap Bergma s boud o the permaet Shearer s Lemma Number
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More informationSequences and Series of Functions
Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges
More informationEntropy Rates and Asymptotic Equipartition
Chapter 29 Etropy Rates ad Asymptotic Equipartitio Sectio 29. itroduces the etropy rate the asymptotic etropy per time-step of a stochastic process ad shows that it is well-defied; ad similarly for iformatio,
More informationAdvanced Stochastic Processes.
Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.
More informationDistribution of Random Samples & Limit theorems
STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to
More informationLecture 19: Convergence
Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may
More information62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 +
62. Power series Defiitio 16. (Power series) Give a sequece {c }, the series c x = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + is called a power series i the variable x. The umbers c are called the coefficiets of
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013 Fuctioal Law of Large Numbers. Costructio of the Wieer Measure Cotet. 1. Additioal techical results o weak covergece
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 6 9/24/2008 DISCRETE RANDOM VARIABLES AND THEIR EXPECTATIONS
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 6 9/24/2008 DISCRETE RANDOM VARIABLES AND THEIR EXPECTATIONS Cotets 1. A few useful discrete radom variables 2. Joit, margial, ad
More informationLaw of the sum of Bernoulli random variables
Law of the sum of Beroulli radom variables Nicolas Chevallier Uiversité de Haute Alsace, 4, rue des frères Lumière 68093 Mulhouse icolas.chevallier@uha.fr December 006 Abstract Let be the set of all possible
More informationLecture 8: Convergence of transformations and law of large numbers
Lecture 8: Covergece of trasformatios ad law of large umbers Trasformatio ad covergece Trasformatio is a importat tool i statistics. If X coverges to X i some sese, we ofte eed to check whether g(x ) coverges
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS
MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak
More informationSTAT Homework 1 - Solutions
STAT-36700 Homework 1 - Solutios Fall 018 September 11, 018 This cotais solutios for Homework 1. Please ote that we have icluded several additioal commets ad approaches to the problems to give you better
More informationIntegrable Functions. { f n } is called a determining sequence for f. If f is integrable with respect to, then f d does exist as a finite real number
MATH 532 Itegrable Fuctios Dr. Neal, WKU We ow shall defie what it meas for a measurable fuctio to be itegrable, show that all itegral properties of simple fuctios still hold, ad the give some coditios
More information2.4 Sequences, Sequences of Sets
72 CHAPTER 2. IMPORTANT PROPERTIES OF R 2.4 Sequeces, Sequeces of Sets 2.4.1 Sequeces Defiitio 2.4.1 (sequece Let S R. 1. A sequece i S is a fuctio f : K S where K = { N : 0 for some 0 N}. 2. For each
More informationDefinition 4.2. (a) A sequence {x n } in a Banach space X is a basis for X if. unique scalars a n (x) such that x = n. a n (x) x n. (4.
4. BASES I BAACH SPACES 39 4. BASES I BAACH SPACES Sice a Baach space X is a vector space, it must possess a Hamel, or vector space, basis, i.e., a subset {x γ } γ Γ whose fiite liear spa is all of X ad
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More informationInfinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
More informationNotes 19 : Martingale CLT
Notes 9 : Martigale CLT Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: [Bil95, Chapter 35], [Roc, Chapter 3]. Sice we have ot ecoutered weak covergece i some time, we first recall
More informationECE 901 Lecture 14: Maximum Likelihood Estimation and Complexity Regularization
ECE 90 Lecture 4: Maximum Likelihood Estimatio ad Complexity Regularizatio R Nowak 5/7/009 Review : Maximum Likelihood Estimatio We have iid observatios draw from a ukow distributio Y i iid p θ, i,, where
More information5.1 A mutual information bound based on metric entropy
Chapter 5 Global Fao Method I this chapter, we exted the techiques of Chapter 2.4 o Fao s method the local Fao method) to a more global costructio. I particular, we show that, rather tha costructig a local
More informationOptimally Sparse SVMs
A. Proof of Lemma 3. We here prove a lower boud o the umber of support vectors to achieve geeralizatio bouds of the form which we cosider. Importatly, this result holds ot oly for liear classifiers, but
More informationShannon s noiseless coding theorem
18.310 lecture otes May 4, 2015 Shao s oiseless codig theorem Lecturer: Michel Goemas I these otes we discuss Shao s oiseless codig theorem, which is oe of the foudig results of the field of iformatio
More informationEE 4TM4: Digital Communications II Information Measures
EE 4TM4: Digital Commuicatios II Iformatio Measures Defiitio : The etropy H(X) of a discrete radom variable X is defied by We also write H(p) for the above quatity. Lemma : H(X) 0. H(X) = x X Proof: 0
More informationInformation Theory and Statistics Lecture 4: Lempel-Ziv code
Iformatio Theory ad Statistics Lecture 4: Lempel-Ziv code Łukasz Dębowski ldebowsk@ipipa.waw.pl Ph. D. Programme 203/204 Etropy rate is the limitig compressio rate Theorem For a statioary process (X i)
More informationChapter IV Integration Theory
Chapter IV Itegratio Theory Lectures 32-33 1. Costructio of the itegral I this sectio we costruct the abstract itegral. As a matter of termiology, we defie a measure space as beig a triple (, A, µ), where
More informationMAT1026 Calculus II Basic Convergence Tests for Series
MAT026 Calculus II Basic Covergece Tests for Series Egi MERMUT 202.03.08 Dokuz Eylül Uiversity Faculty of Sciece Departmet of Mathematics İzmir/TURKEY Cotets Mootoe Covergece Theorem 2 2 Series of Real
More informationIt is always the case that unions, intersections, complements, and set differences are preserved by the inverse image of a function.
MATH 532 Measurable Fuctios Dr. Neal, WKU Throughout, let ( X, F, µ) be a measure space ad let (!, F, P ) deote the special case of a probability space. We shall ow begi to study real-valued fuctios defied
More informationMaximum Likelihood Estimation and Complexity Regularization
ECE90 Sprig 004 Statistical Regularizatio ad Learig Theory Lecture: 4 Maximum Likelihood Estimatio ad Complexity Regularizatio Lecturer: Rob Nowak Scribe: Pam Limpiti Review : Maximum Likelihood Estimatio
More information1 Convergence in Probability and the Weak Law of Large Numbers
36-752 Advaced Probability Overview Sprig 2018 8. Covergece Cocepts: i Probability, i L p ad Almost Surely Istructor: Alessadro Rialdo Associated readig: Sec 2.4, 2.5, ad 4.11 of Ash ad Doléas-Dade; Sec
More informationOn Random Line Segments in the Unit Square
O Radom Lie Segmets i the Uit Square Thomas A. Courtade Departmet of Electrical Egieerig Uiversity of Califoria Los Ageles, Califoria 90095 Email: tacourta@ee.ucla.edu I. INTRODUCTION Let Q = [0, 1] [0,
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More informationLecture 3 : Random variables and their distributions
Lecture 3 : Radom variables ad their distributios 3.1 Radom variables Let (Ω, F) ad (S, S) be two measurable spaces. A map X : Ω S is measurable or a radom variable (deoted r.v.) if X 1 (A) {ω : X(ω) A}
More informationAn Introduction to Randomized Algorithms
A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 6 9/23/2013. Brownian motion. Introduction
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 6 9/23/203 Browia motio. Itroductio Cotet.. A heuristic costructio of a Browia motio from a radom walk. 2. Defiitio ad basic properties
More information6.3 Testing Series With Positive Terms
6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial
More informationMath 61CM - Solutions to homework 3
Math 6CM - Solutios to homework 3 Cédric De Groote October 2 th, 208 Problem : Let F be a field, m 0 a fixed oegative iteger ad let V = {a 0 + a x + + a m x m a 0,, a m F} be the vector space cosistig
More informationRefinement of Two Fundamental Tools in Information Theory
Refiemet of Two Fudametal Tools i Iformatio Theory Raymod W. Yeug Istitute of Network Codig The Chiese Uiversity of Hog Kog Joit work with Siu Wai Ho ad Sergio Verdu Discotiuity of Shao s Iformatio Measures
More informationCEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering
CEE 5 Autum 005 Ucertaity Cocepts for Geotechical Egieerig Basic Termiology Set A set is a collectio of (mutually exclusive) objects or evets. The sample space is the (collectively exhaustive) collectio
More informationSequences A sequence of numbers is a function whose domain is the positive integers. We can see that the sequence
Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece 1, 1, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet
More informationBasics of Probability Theory (for Theory of Computation courses)
Basics of Probability Theory (for Theory of Computatio courses) Oded Goldreich Departmet of Computer Sciece Weizma Istitute of Sciece Rehovot, Israel. oded.goldreich@weizma.ac.il November 24, 2008 Preface.
More informationLinear Regression Demystified
Liear Regressio Demystified Liear regressio is a importat subject i statistics. I elemetary statistics courses, formulae related to liear regressio are ofte stated without derivatio. This ote iteds to
More informationLecture 3 The Lebesgue Integral
Lecture 3: The Lebesgue Itegral 1 of 14 Course: Theory of Probability I Term: Fall 2013 Istructor: Gorda Zitkovic Lecture 3 The Lebesgue Itegral The costructio of the itegral Uless expressly specified
More informationRademacher Complexity
EECS 598: Statistical Learig Theory, Witer 204 Topic 0 Rademacher Complexity Lecturer: Clayto Scott Scribe: Ya Deg, Kevi Moo Disclaimer: These otes have ot bee subjected to the usual scrutiy reserved for
More informationLecture 7: October 18, 2017
Iformatio ad Codig Theory Autum 207 Lecturer: Madhur Tulsiai Lecture 7: October 8, 207 Biary hypothesis testig I this lecture, we apply the tools developed i the past few lectures to uderstad the problem
More informationON MEAN ERGODIC CONVERGENCE IN THE CALKIN ALGEBRAS
PROCEEDINGS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 00, Number 0, Pages 000 000 S 0002-9939(XX0000-0 ON MEAN ERGODIC CONVERGENCE IN THE CALKIN ALGEBRAS MARCH T. BOEDIHARDJO AND WILLIAM B. JOHNSON 2
More informationEconomics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator
Ecoomics 24B Relatio to Method of Momets ad Maximum Likelihood OLSE as a Maximum Likelihood Estimator Uder Assumptio 5 we have speci ed the distributio of the error, so we ca estimate the model parameters
More informationEmpirical Process Theory and Oracle Inequalities
Stat 928: Statistical Learig Theory Lecture: 10 Empirical Process Theory ad Oracle Iequalities Istructor: Sham Kakade 1 Risk vs Risk See Lecture 0 for a discussio o termiology. 2 The Uio Boud / Boferoi
More informationMath 155 (Lecture 3)
Math 55 (Lecture 3) September 8, I this lecture, we ll cosider the aswer to oe of the most basic coutig problems i combiatorics Questio How may ways are there to choose a -elemet subset of the set {,,,
More informationProbability 2 - Notes 10. Lemma. If X is a random variable and g(x) 0 for all x in the support of f X, then P(g(X) 1) E[g(X)].
Probability 2 - Notes 0 Some Useful Iequalities. Lemma. If X is a radom variable ad g(x 0 for all x i the support of f X, the P(g(X E[g(X]. Proof. (cotiuous case P(g(X Corollaries x:g(x f X (xdx x:g(x
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationInequalities for Entropies of Sets of Subsets of Random Variables
Iequalities for Etropies of Sets of Subsets of Radom Variables Chao Tia AT&T Labs-Research Florham Par, NJ 0792, USA. tia@research.att.com Abstract Ha s iequality o the etropy rates of subsets of radom
More informationLecture 12: November 13, 2018
Mathematical Toolkit Autum 2018 Lecturer: Madhur Tulsiai Lecture 12: November 13, 2018 1 Radomized polyomial idetity testig We will use our kowledge of coditioal probability to prove the followig lemma,
More informationOutput Analysis and Run-Length Control
IEOR E4703: Mote Carlo Simulatio Columbia Uiversity c 2017 by Marti Haugh Output Aalysis ad Ru-Legth Cotrol I these otes we describe how the Cetral Limit Theorem ca be used to costruct approximate (1 α%
More information17. Joint distributions of extreme order statistics Lehmann 5.1; Ferguson 15
17. Joit distributios of extreme order statistics Lehma 5.1; Ferguso 15 I Example 10., we derived the asymptotic distributio of the maximum from a radom sample from a uiform distributio. We did this usig
More informationON POINTWISE BINOMIAL APPROXIMATION
Iteratioal Joural of Pure ad Applied Mathematics Volume 71 No. 1 2011, 57-66 ON POINTWISE BINOMIAL APPROXIMATION BY w-functions K. Teerapabolar 1, P. Wogkasem 2 Departmet of Mathematics Faculty of Sciece
More information4.3 Growth Rates of Solutions to Recurrences
4.3. GROWTH RATES OF SOLUTIONS TO RECURRENCES 81 4.3 Growth Rates of Solutios to Recurreces 4.3.1 Divide ad Coquer Algorithms Oe of the most basic ad powerful algorithmic techiques is divide ad coquer.
More informationECE 901 Lecture 12: Complexity Regularization and the Squared Loss
ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality
More informationST5215: Advanced Statistical Theory
ST525: Advaced Statistical Theory Departmet of Statistics & Applied Probability Tuesday, September 7, 2 ST525: Advaced Statistical Theory Lecture : The law of large umbers The Law of Large Numbers The
More informationA sequence of numbers is a function whose domain is the positive integers. We can see that the sequence
Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece,, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet as
More information1 Approximating Integrals using Taylor Polynomials
Seughee Ye Ma 8: Week 7 Nov Week 7 Summary This week, we will lear how we ca approximate itegrals usig Taylor series ad umerical methods. Topics Page Approximatig Itegrals usig Taylor Polyomials. Defiitios................................................
More informationSieve Estimators: Consistency and Rates of Convergence
EECS 598: Statistical Learig Theory, Witer 2014 Topic 6 Sieve Estimators: Cosistecy ad Rates of Covergece Lecturer: Clayto Scott Scribe: Julia Katz-Samuels, Brado Oselio, Pi-Yu Che Disclaimer: These otes
More informationAn Introduction to Asymptotic Theory
A Itroductio to Asymptotic Theory Pig Yu School of Ecoomics ad Fiace The Uiversity of Hog Kog Pig Yu (HKU) Asymptotic Theory 1 / 20 Five Weapos i Asymptotic Theory Five Weapos i Asymptotic Theory Pig Yu
More informationSequences. Notation. Convergence of a Sequence
Sequeces A sequece is essetially just a list. Defiitio (Sequece of Real Numbers). A sequece of real umbers is a fuctio Z (, ) R for some real umber. Do t let the descriptio of the domai cofuse you; it
More informationQuick Review of Probability
Quick Review of Probability Berli Che Departmet of Computer Sciece & Iformatio Egieerig Natioal Taiwa Normal Uiversity Refereces: 1. W. Navidi. Statistics for Egieerig ad Scietists. Chapter 2 & Teachig
More informationINFINITE SEQUENCES AND SERIES
11 INFINITE SEQUENCES AND SERIES INFINITE SEQUENCES AND SERIES 11.4 The Compariso Tests I this sectio, we will lear: How to fid the value of a series by comparig it with a kow series. COMPARISON TESTS
More informationAsymptotic distribution of products of sums of independent random variables
Proc. Idia Acad. Sci. Math. Sci. Vol. 3, No., May 03, pp. 83 9. c Idia Academy of Scieces Asymptotic distributio of products of sums of idepedet radom variables YANLING WANG, SUXIA YAO ad HONGXIA DU ollege
More informationLecture 3: August 31
36-705: Itermediate Statistics Fall 018 Lecturer: Siva Balakrisha Lecture 3: August 31 This lecture will be mostly a summary of other useful expoetial tail bouds We will ot prove ay of these i lecture,
More informationLecture Notes for Analysis Class
Lecture Notes for Aalysis Class Topological Spaces A topology for a set X is a collectio T of subsets of X such that: (a) X ad the empty set are i T (b) Uios of elemets of T are i T (c) Fiite itersectios
More informationLecture 11: Channel Coding Theorem: Converse Part
EE376A/STATS376A Iformatio Theory Lecture - 02/3/208 Lecture : Chael Codig Theorem: Coverse Part Lecturer: Tsachy Weissma Scribe: Erdem Bıyık I this lecture, we will cotiue our discussio o chael codig
More informationSTA Object Data Analysis - A List of Projects. January 18, 2018
STA 6557 Jauary 8, 208 Object Data Aalysis - A List of Projects. Schoeberg Mea glaucomatous shape chages of the Optic Nerve Head regio i aimal models 2. Aalysis of VW- Kedall ati-mea shapes with a applicatio
More informationSolution. 1 Solutions of Homework 1. Sangchul Lee. October 27, Problem 1.1
Solutio Sagchul Lee October 7, 017 1 Solutios of Homework 1 Problem 1.1 Let Ω,F,P) be a probability space. Show that if {A : N} F such that A := lim A exists, the PA) = lim PA ). Proof. Usig the cotiuity
More information4. Partial Sums and the Central Limit Theorem
1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems
More informationCHAPTER 10 INFINITE SEQUENCES AND SERIES
CHAPTER 10 INFINITE SEQUENCES AND SERIES 10.1 Sequeces 10.2 Ifiite Series 10.3 The Itegral Tests 10.4 Compariso Tests 10.5 The Ratio ad Root Tests 10.6 Alteratig Series: Absolute ad Coditioal Covergece
More informationn outcome is (+1,+1, 1,..., 1). Let the r.v. X denote our position (relative to our starting point 0) after n moves. Thus X = X 1 + X 2 + +X n,
CS 70 Discrete Mathematics for CS Sprig 2008 David Wager Note 9 Variace Questio: At each time step, I flip a fair coi. If it comes up Heads, I walk oe step to the right; if it comes up Tails, I walk oe
More informationAddition: Property Name Property Description Examples. a+b = b+a. a+(b+c) = (a+b)+c
Notes for March 31 Fields: A field is a set of umbers with two (biary) operatios (usually called additio [+] ad multiplicatio [ ]) such that the followig properties hold: Additio: Name Descriptio Commutativity
More informationSeunghee Ye Ma 8: Week 5 Oct 28
Week 5 Summary I Sectio, we go over the Mea Value Theorem ad its applicatios. I Sectio 2, we will recap what we have covered so far this term. Topics Page Mea Value Theorem. Applicatios of the Mea Value
More informationThe multiplicative structure of finite field and a construction of LRC
IERG6120 Codig for Distributed Storage Systems Lecture 8-06/10/2016 The multiplicative structure of fiite field ad a costructio of LRC Lecturer: Keeth Shum Scribe: Zhouyi Hu Notatios: We use the otatio
More information11.5 Alternating Series, Absolute and Conditional Convergence
.5.5 Alteratig Series, Absolute ad Coditioal Covergece We have see that the harmoic series diverges. It may come as a surprise the to lear that ) 2 + 3 4 + + )+ + = ) + coverges. To see this, let s be
More informationIntroduction to Extreme Value Theory Laurens de Haan, ISM Japan, Erasmus University Rotterdam, NL University of Lisbon, PT
Itroductio to Extreme Value Theory Laures de Haa, ISM Japa, 202 Itroductio to Extreme Value Theory Laures de Haa Erasmus Uiversity Rotterdam, NL Uiversity of Lisbo, PT Itroductio to Extreme Value Theory
More informationGeneralized Semi- Markov Processes (GSMP)
Geeralized Semi- Markov Processes (GSMP) Summary Some Defiitios Markov ad Semi-Markov Processes The Poisso Process Properties of the Poisso Process Iterarrival times Memoryless property ad the residual
More informationChapter 5. Inequalities. 5.1 The Markov and Chebyshev inequalities
Chapter 5 Iequalities 5.1 The Markov ad Chebyshev iequalities As you have probably see o today s frot page: every perso i the upper teth percetile ears at least 1 times more tha the average salary. I other
More informationKernel density estimator
Jauary, 07 NONPARAMETRIC ERNEL DENSITY ESTIMATION I this lecture, we discuss kerel estimatio of probability desity fuctios PDF Noparametric desity estimatio is oe of the cetral problems i statistics I
More informationDiscrete Mathematics for CS Spring 2008 David Wagner Note 22
CS 70 Discrete Mathematics for CS Sprig 2008 David Wager Note 22 I.I.D. Radom Variables Estimatig the bias of a coi Questio: We wat to estimate the proportio p of Democrats i the US populatio, by takig
More informationREGRESSION WITH QUADRATIC LOSS
REGRESSION WITH QUADRATIC LOSS MAXIM RAGINSKY Regressio with quadratic loss is aother basic problem studied i statistical learig theory. We have a radom couple Z = X, Y ), where, as before, X is a R d
More informationB Supplemental Notes 2 Hypergeometric, Binomial, Poisson and Multinomial Random Variables and Borel Sets
B671-672 Supplemetal otes 2 Hypergeometric, Biomial, Poisso ad Multiomial Radom Variables ad Borel Sets 1 Biomial Approximatio to the Hypergeometric Recall that the Hypergeometric istributio is fx = x
More informationJournal of Multivariate Analysis. Superefficient estimation of the marginals by exploiting knowledge on the copula
Joural of Multivariate Aalysis 102 (2011) 1315 1319 Cotets lists available at ScieceDirect Joural of Multivariate Aalysis joural homepage: www.elsevier.com/locate/jmva Superefficiet estimatio of the margials
More information32 estimating the cumulative distribution function
32 estimatig the cumulative distributio fuctio 4.6 types of cofidece itervals/bads Let F be a class of distributio fuctios F ad let θ be some quatity of iterest, such as the mea of F or the whole fuctio
More informationQuick Review of Probability
Quick Review of Probability Berli Che Departmet of Computer Sciece & Iformatio Egieerig Natioal Taiwa Normal Uiversity Refereces: 1. W. Navidi. Statistics for Egieerig ad Scietists. Chapter & Teachig Material.
More information1 Review and Overview
DRAFT a fial versio will be posted shortly CS229T/STATS231: Statistical Learig Theory Lecturer: Tegyu Ma Lecture #3 Scribe: Migda Qiao October 1, 2013 1 Review ad Overview I the first half of this course,
More informationThe log-behavior of n p(n) and n p(n)/n
Ramauja J. 44 017, 81-99 The log-behavior of p ad p/ William Y.C. Che 1 ad Ke Y. Zheg 1 Ceter for Applied Mathematics Tiaji Uiversity Tiaji 0007, P. R. Chia Ceter for Combiatorics, LPMC Nakai Uivercity
More informationLecture 9: Expanders Part 2, Extractors
Lecture 9: Expaders Part, Extractors Topics i Complexity Theory ad Pseudoradomess Sprig 013 Rutgers Uiversity Swastik Kopparty Scribes: Jaso Perry, Joh Kim I this lecture, we will discuss further the pseudoradomess
More informationECE 901 Lecture 13: Maximum Likelihood Estimation
ECE 90 Lecture 3: Maximum Likelihood Estimatio R. Nowak 5/7/009 The focus of this lecture is to cosider aother approach to learig based o maximum likelihood estimatio. Ulike earlier approaches cosidered
More information