Lecture Notes 5 Convergence and Limit Theorems. Convergence with Probability 1. Convergence in Mean Square. Convergence in Probability, WLLN
|
|
- Madeleine Davidson
- 5 years ago
- Views:
Transcription
1 Lecture Notes 5 Convergence and Limit Theorems Motivation Convergence with Probability Convergence in Mean Square Convergence in Probability, WLLN Convergence in Distribution, CLT EE 278: Convergence and Limit Theorems Page 5
2 Motivation One of the key questions in statistical signal processing is how to estimate the statistics of a r.v., e.g., its mean, variance, distribution, etc. To estimate such a statistic, we collect samples and use an estimator in the form of a sample average How good is the estimator? Does it converge to the true statistic? How many samples do we need to ensure with some confidence that we are within a certain range of the true value of the statistic? Another key question in statistical signal processing is how to estimate a signal from noisy observations, e.g., using MSE or linear MSE Does the estimator converge to the true signal? How many observations do we need to achieve a desired estimation accuracy? The subject of convergence and limit theorems for r.v.s addresses such questions EE 278: Convergence and Limit Theorems Page 5 2
3 Example: Estimating the Mean of a R.V. Let X be a r.v. with finite but unknown mean E(X) To estimate the mean we generate X,X 2,...,X n i.i.d. samples drawn according to the same distribution as X and compute the sample mean S n = n X i n Does S n converge to E(X) as we increase n? If so, how fast? But what does it mean to say that a r.v. sequence S n converges to E(X)? First we give an example: Let X,X 2,...,X n,... be i.i.d. N(,) We use Matlab to generate 6 sets of outcomes of X,...,X n,...,x We then plot s n for the 6 sets of outcomes as a function of n Note that each s n sequence appears to be converging to, the mean of the r.v., as n increases i= EE 278: Convergence and Limit Theorems Page 5 3
4 Plots of Sample Sequences of S n 2 sn sn sn sn n EE 278: Convergence and Limit Theorems Page 5 4
5 Convergence With Probability Recall that a sequence of numbers x,x 2,...,x n,... converges to x if for every ǫ >, there exists an m(ǫ) such that x n x < ǫ for every n m(ǫ) Now consider a sequence of r.v.s X,X 2,...,X n,... all defined on the same probability space Ω. For every ω Ω we obtain a sample sequence (sequence of numbers) X (ω),x 2 (ω),...,x n (ω),... A sequence X,X 2,X 3,... of r.v.s is said to converge to a random variable X with probability (w.p., also called almost surely) if P{ω : lim n X n(ω) = X(ω)} = This means that the set of sample paths that converge to X(ω), in the sense of a sequence converging to a limit, has probability Equivalently, X,X 2,...,X n,... converges w.p. if for every ǫ >, lim P{ X n X < ǫ for every n m} = m EE 278: Convergence and Limit Theorems Page 5 5
6 Example : Let X,X 2,...,X n be i.i.d. Bern(/2), and define Y n = 2 n n i= X i. Show that the sequence Y n converges to w.p. Solution: To show this, let ǫ > (and ǫ < 2 m ), and consider P{ Y n < ǫ for all n m} = P{X n = for some n m} = P{X n = for all n m} = ( 2 )m as m An important example of convergence w.p.: the Strong Law of Large Numbers (SLLN), which says that if X,X 2,...,X n,... are i.i.d. with finite mean E(X), then the sequence of sample means S n E(X) w.p. The previous Matlab example is a good demonstration of the SLLN each of the 6 sample paths appears to be converging to, which is E(X) The proof of the SLLN and other convergence w.p. results are beyond the scope of this course. Take Stats 3 if you want to learn a lot more about this EE 278: Convergence and Limit Theorems Page 5 6
7 Convergence in Mean Square A sequence of r.v.s X,X 2,...,X n,... converges to a random variable X in mean square (m.s.) if Example: Estimating the mean. lim n E[ (X n X) 2] = Let X,X 2,...,X n,... be i.i.d. with finite mean E(X) and variance Var(X). Then S n E(X) in m.s. Proof: Here we need to show that First note that E(S n ) = E [ n lim E[ (S n E(X)) 2] = n n ] X i i= = n n E(X i ) = n i= So, S n is an unbiased estimate of E(X) n E(X) = E(X) i= EE 278: Convergence and Limit Theorems Page 5 7
8 Now to prove convergence in m.s., consider E [ (S n E(X)) 2] = E [ (S n E(S n )) 2] ( = E n X i n n i= ( = n n 2E X i i= ( n ) = n 2Var X i i= ( n ) = Var(X i ) n 2 i= = n 2(nVar(X)) ) 2 n E(X) i= ) 2 n E(X) i= = Var(X) as n n since {X i } are independent EE 278: Convergence and Limit Theorems Page 5 8
9 Note that the proof works even if the r.v.s are only pairwise independent or even only uncorrelated Example: Consider the best linear MSE estimates found in the first estimation example of Lecture Notes 4 as a sequence of r.v.s ˆX, ˆX 2,..., ˆX n,..., where ˆX n is the best linear estimate of X given the first n observations. This sequence converges in m.s. to X since MSE n Convergence in m.s. does not necessarily imply convergence w.p. Example 2: Let X,X 2,...,X n,... be a sequence of independent r.v.s such that with probability n X n = with probability n Clearly this sequence converges to in m.s., but does it converge w.p.? EE 278: Convergence and Limit Theorems Page 5 9
10 It actually does not, since for < ǫ < and any m n ( P{ X n < ǫ for all n m} = lim ) n i = lim n = lim n = lim n i=m n i=m ( i Also convergence w.p. does not imply convergence in m.s. Consider the sequence in Example. Since E [ (Y n ) 2] = ( 2) n2 2n = 2 n, i ) (m ) (m+) (n ) m m n m n the sequence does not converge in m.s. even though it converges w.p. EE 278: Convergence and Limit Theorems Page 5
11 Example: Convergence to a random variable: Flip a coin with random bias P conditionally independently to obtain the sequence X,X 2,...,X n,..., where as usual X i = if the ith coin flip is heads and X i = otherwise As we already know, the r.v.s X,X 2,...,X n are not independent, but given P = p they are i.i.d. Bern(p) It is easy to show using iterated expectation that E(S n ) = E(X ) = E(P) In a homework exercise, you will show that S n P (not to E(P)) in m.s. EE 278: Convergence and Limit Theorems Page 5
12 Convergence in Probability A sequence of r.v.s X,X 2,...,X n,... converges to a r.v. X in probability if for any ǫ >, lim n P{ X n X < ǫ} = Convergence w.p. implies convergence in probability. The converse is not necessarily true, so convergence w.p. is stronger than in probability Example 3: Let X,X 2,...,X n,... be independent such that { with probability n X n = n with probability n Clearly, this sequence converges in probability to, since P{ X n > ǫ} = P{X n > ǫ} = n as n But does it converge w.p.? The answer is no (see Example 2) EE 278: Convergence and Limit Theorems Page 5 2
13 Convergence in m.s. implies convergence in probability. To show this we use the Markov inequality. For any ǫ >, P{ X n X > ǫ} = P{(X n X) 2 > ǫ 2 } E[(X n X) 2 ] ǫ 2 If X n X in m.s., then lim n E[ (X n X) 2] = lim n P{ X n X > ǫ} =, i.e., X n X in probability The converse is not necessarily true. In Example 3, X n converges in probability. Now consider E [ (X n ) 2] ( = ) +n 2 n n = n as n Thus X n does not converge in m.s. So convergence in probability is weaker than both convergence w.p. and in m.s. EE 278: Convergence and Limit Theorems Page 5 3
14 The Weak Law of Large Numbers The WLLN states that if X,X 2,...,X n,... is a sequence of i.i.d. r.v.s with finite mean E(X) and variance Var(X), then S n = n X i E(X) in probability n i= We already proved that S n E(X) in m.s., and since convergence in m.s. implies convergence in probability, S n E(X) in probability So, WLLN requires only uncorrelation of the r.v.s (SLLN requires independence) EE 278: Convergence and Limit Theorems Page 5 4
15 Confidence Intervals Given ǫ,δ >, how large should n, the number of samples, be so that P{ S n E(X) ǫ} δ, i.e., S n is within ±ǫ of E(X) with probability δ? Let s use the Chebyshev inequality: P{ S n E(X) ǫ} = P{ S n E(S n ) ǫ} Var(S n) ǫ 2 = Var(X) nǫ 2 So n should be large enough that: Var(X)/nǫ 2 δ n Var(X)/δǫ 2 Example: Let ǫ =.σ X and δ =.. The number of samples should satisfy n σ 2 X..σ 2 X = 5, i.e., 5 samples ensure that S n is within ±.σ X of E(X) with probability.999, independent of the distribution of X EE 278: Convergence and Limit Theorems Page 5 5
16 Convergence in Distribution A sequence of r.v.s X,X 2,...,X n,... converges in distribution to a r.v. X if lim n F X n (x) = F X (x) for every x at which F X (x) is continuous Convergence in probability implies convergence in distribution so convergence in distribution is the weakest form of convergence we discuss The most important example of convergence in distribution is the Central Limit Theorem (CLT). Let X,X 2,...,X n,... be i.i.d. r.v.s with finite mean E(X) and variance σx 2. Consider the normalized sum Z n = n n i= X i E(X) The sum is called normalized because E(Z n ) = and Var(Z n ) = The Central Limit Theorem states that Z n Z N(,) in distribution, i.e., { Q(z) z lim F Z n n (z) = Φ(z) = Q( z) z < σ X EE 278: Convergence and Limit Theorems Page 5 6
17 Example: Let X,X 2,... be i.i.d. U[,] r.v.s. The normalized sum is Z n = n i= X i/ n/3. The following plots show the pdf of Z n for n =,2,4,6. Note how quickly the pdf of Z n approaches the Gaussian pdf Z4 pdf Z2 pdf Z pdf Z6 pdf z EE 278: Convergence and Limit Theorems Page 5 7
18 Example: Let X,X 2,... be i.i.d. Bern(/2). The normalized sum is Z n = n i= (X i.5)/ n/4. The following plots show the cdf of Z n for n =,2,6. Z n is discrete and thus has no pdf, but its cdf converges to the Gaussian cdf Z cdf Z2 cdf Z6 cdf z EE 278: Convergence and Limit Theorems Page 5 8
19 Application: Confidence Intervals Let X,X 2,...,X n be i.i.d. with finite mean E(X) and variance Var(X) and let S n be the sample mean Given ǫ, δ >, how large should n, the number of samples, be so that P{ S n E(X) ǫ} δ? We can use the CLT to find an estimate of n as follows: { n } P{ S n E(S n ) ǫ} = P (X i E(X)) ǫ n i= { n } = P (X i E(X)) ǫ n σ X n σ X i= ( ) ǫ n 2Q Example: For ǫ =.σ X, δ =., set 2Q(. n) =., so. n = 3.3 or n = 89 much smaller than n 5 obtained by the Chebyshev inequality σ X EE 278: Convergence and Limit Theorems Page 5 9
20 CLT for Random Vectors The CLT applies to i.i.d. sequences of random vectors Let X,X 2,...,X n,... be a sequence of i.i.d. k-dimensional random vectors with finite mean µ and nonsingular covariance matrix Σ. Define the sequence of random vectors Z,Z 2,...,Z n,... by Z n = n (X i µ) n The Central Limit Theorem for random vectors states that as n i= Z n Z N(,Σ) in distribution Example: Let X,X 2,...,X n,... be a sequence of i.i.d. 2-dimensional random vectors with { x +x 2 < x <, < x 2 < f X (x,x 2 ) = otherwise The following plots show the joint pdf of Y n = n i= X i for n =,2,3,4. Note how quickly it looks Gaussian. EE 278: Convergence and Limit Theorems Page 5 2
21 2 Y pdf.5.5 Y2 pdf Y3 pdf Y4 pdf EE 278: Convergence and Limit Theorems Page 5 2
22 Relationships Between Types of Convergence The following figure summarizes the relationships between the different types of convergence we discussed with probability in probability in distribution in mean square EE 278: Convergence and Limit Theorems Page 5 22
CS145: Probability & Computing
CS45: Probability & Computing Lecture 5: Concentration Inequalities, Law of Large Numbers, Central Limit Theorem Instructor: Eli Upfal Brown University Computer Science Figure credits: Bertsekas & Tsitsiklis,
More informationSolutions to Homework Set #6 (Prepared by Lele Wang)
Solutions to Homework Set #6 (Prepared by Lele Wang) Gaussian random vector Given a Gaussian random vector X N (µ, Σ), where µ ( 5 ) T and 0 Σ 4 0 0 0 9 (a) Find the pdfs of i X, ii X + X 3, iii X + X
More informationLecture Notes 3 Convergence (Chapter 5)
Lecture Notes 3 Convergence (Chapter 5) 1 Convergence of Random Variables Let X 1, X 2,... be a sequence of random variables and let X be another random variable. Let F n denote the cdf of X n and let
More informationLecture 7: Chapter 7. Sums of Random Variables and Long-Term Averages
Lecture 7: Chapter 7. Sums of Random Variables and Long-Term Averages ELEC206 Probability and Random Processes, Fall 2014 Gil-Jin Jang gjang@knu.ac.kr School of EE, KNU page 1 / 15 Chapter 7. Sums of Random
More informationLecture 2: Repetition of probability theory and statistics
Algorithms for Uncertainty Quantification SS8, IN2345 Tobias Neckel Scientific Computing in Computer Science TUM Lecture 2: Repetition of probability theory and statistics Concept of Building Block: Prerequisites:
More informationCOMPSCI 240: Reasoning Under Uncertainty
COMPSCI 240: Reasoning Under Uncertainty Andrew Lan and Nic Herndon University of Massachusetts at Amherst Spring 2019 Lecture 20: Central limit theorem & The strong law of large numbers Markov and Chebyshev
More informationProving the central limit theorem
SOR3012: Stochastic Processes Proving the central limit theorem Gareth Tribello March 3, 2019 1 Purpose In the lectures and exercises we have learnt about the law of large numbers and the central limit
More informationLecture 4: September Reminder: convergence of sequences
36-705: Intermediate Statistics Fall 2017 Lecturer: Siva Balakrishnan Lecture 4: September 6 In this lecture we discuss the convergence of random variables. At a high-level, our first few lectures focused
More informationUniversity of Regina. Lecture Notes. Michael Kozdron
University of Regina Statistics 252 Mathematical Statistics Lecture Notes Winter 2005 Michael Kozdron kozdron@math.uregina.ca www.math.uregina.ca/ kozdron Contents 1 The Basic Idea of Statistics: Estimating
More information8 Laws of large numbers
8 Laws of large numbers 8.1 Introduction We first start with the idea of standardizing a random variable. Let X be a random variable with mean µ and variance σ 2. Then Z = (X µ)/σ will be a random variable
More informationLecture 8. October 22, Department of Biostatistics Johns Hopkins Bloomberg School of Public Health Johns Hopkins University.
Lecture 8 Department of Biostatistics Johns Hopkins Bloomberg School of Public Health Johns Hopkins University October 22, 2007 1 2 3 4 5 6 1 Define convergent series 2 Define the Law of Large Numbers
More informationSTA205 Probability: Week 8 R. Wolpert
INFINITE COIN-TOSS AND THE LAWS OF LARGE NUMBERS The traditional interpretation of the probability of an event E is its asymptotic frequency: the limit as n of the fraction of n repeated, similar, and
More informationMAS113 Introduction to Probability and Statistics
MAS113 Introduction to Probability and Statistics School of Mathematics and Statistics, University of Sheffield 2018 19 Identically distributed Suppose we have n random variables X 1, X 2,..., X n. Identically
More informationAppendix A : Introduction to Probability and stochastic processes
A-1 Mathematical methods in communication July 5th, 2009 Appendix A : Introduction to Probability and stochastic processes Lecturer: Haim Permuter Scribe: Shai Shapira and Uri Livnat The probability of
More informationQuick Tour of Basic Probability Theory and Linear Algebra
Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra CS224w: Social and Information Network Analysis Fall 2011 Quick Tour of and Linear Algebra Quick Tour of and Linear Algebra Outline Definitions
More informationLecture Notes 7 Stationary Random Processes. Strict-Sense and Wide-Sense Stationarity. Autocorrelation Function of a Stationary Process
Lecture Notes 7 Stationary Random Processes Strict-Sense and Wide-Sense Stationarity Autocorrelation Function of a Stationary Process Power Spectral Density Continuity and Integration of Random Processes
More informationEstimation theory. Parametric estimation. Properties of estimators. Minimum variance estimator. Cramer-Rao bound. Maximum likelihood estimators
Estimation theory Parametric estimation Properties of estimators Minimum variance estimator Cramer-Rao bound Maximum likelihood estimators Confidence intervals Bayesian estimation 1 Random Variables Let
More informationAlgorithms for Uncertainty Quantification
Algorithms for Uncertainty Quantification Tobias Neckel, Ionuț-Gabriel Farcaș Lehrstuhl Informatik V Summer Semester 2017 Lecture 2: Repetition of probability theory and statistics Example: coin flip Example
More information6.1 Moment Generating and Characteristic Functions
Chapter 6 Limit Theorems The power statistics can mostly be seen when there is a large collection of data points and we are interested in understanding the macro state of the system, e.g., the average,
More information1 Presessional Probability
1 Presessional Probability Probability theory is essential for the development of mathematical models in finance, because of the randomness nature of price fluctuations in the markets. This presessional
More informationRandom Variables. Saravanan Vijayakumaran Department of Electrical Engineering Indian Institute of Technology Bombay
1 / 13 Random Variables Saravanan Vijayakumaran sarva@ee.iitb.ac.in Department of Electrical Engineering Indian Institute of Technology Bombay August 8, 2013 2 / 13 Random Variable Definition A real-valued
More informationIntroduction to Probability
LECTURE NOTES Course 6.041-6.431 M.I.T. FALL 2000 Introduction to Probability Dimitri P. Bertsekas and John N. Tsitsiklis Professors of Electrical Engineering and Computer Science Massachusetts Institute
More informationLimiting Distributions
Limiting Distributions We introduce the mode of convergence for a sequence of random variables, and discuss the convergence in probability and in distribution. The concept of convergence leads us to the
More informationLecture 10: Probability distributions TUESDAY, FEBRUARY 19, 2019
Lecture 10: Probability distributions DANIEL WELLER TUESDAY, FEBRUARY 19, 2019 Agenda What is probability? (again) Describing probabilities (distributions) Understanding probabilities (expectation) Partial
More informationLecture 2: Review of Basic Probability Theory
ECE 830 Fall 2010 Statistical Signal Processing instructor: R. Nowak, scribe: R. Nowak Lecture 2: Review of Basic Probability Theory Probabilistic models will be used throughout the course to represent
More informationE X A M. Probability Theory and Stochastic Processes Date: December 13, 2016 Duration: 4 hours. Number of pages incl.
E X A M Course code: Course name: Number of pages incl. front page: 6 MA430-G Probability Theory and Stochastic Processes Date: December 13, 2016 Duration: 4 hours Resources allowed: Notes: Pocket calculator,
More informationEconomics 583: Econometric Theory I A Primer on Asymptotics
Economics 583: Econometric Theory I A Primer on Asymptotics Eric Zivot January 14, 2013 The two main concepts in asymptotic theory that we will use are Consistency Asymptotic Normality Intuition consistency:
More informationPractice Problem - Skewness of Bernoulli Random Variable. Lecture 7: Joint Distributions and the Law of Large Numbers. Joint Distributions - Example
A little more E(X Practice Problem - Skewness of Bernoulli Random Variable Lecture 7: and the Law of Large Numbers Sta30/Mth30 Colin Rundel February 7, 014 Let X Bern(p We have shown that E(X = p Var(X
More informationPart IA Probability. Definitions. Based on lectures by R. Weber Notes taken by Dexter Chua. Lent 2015
Part IA Probability Definitions Based on lectures by R. Weber Notes taken by Dexter Chua Lent 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after lectures.
More informationLecture 1: August 28
36-705: Intermediate Statistics Fall 2017 Lecturer: Siva Balakrishnan Lecture 1: August 28 Our broad goal for the first few lectures is to try to understand the behaviour of sums of independent random
More informationUCSD ECE153 Handout #30 Prof. Young-Han Kim Thursday, May 15, Homework Set #6 Due: Thursday, May 22, 2011
UCSD ECE153 Handout #30 Prof. Young-Han Kim Thursday, May 15, 2014 Homework Set #6 Due: Thursday, May 22, 2011 1. Linear estimator. Consider a channel with the observation Y = XZ, where the signal X and
More informationProbability Review. Yutian Li. January 18, Stanford University. Yutian Li (Stanford University) Probability Review January 18, / 27
Probability Review Yutian Li Stanford University January 18, 2018 Yutian Li (Stanford University) Probability Review January 18, 2018 1 / 27 Outline 1 Elements of probability 2 Random variables 3 Multiple
More information6 The normal distribution, the central limit theorem and random samples
6 The normal distribution, the central limit theorem and random samples 6.1 The normal distribution We mentioned the normal (or Gaussian) distribution in Chapter 4. It has density f X (x) = 1 σ 1 2π e
More informationChapter 2: Fundamentals of Statistics Lecture 15: Models and statistics
Chapter 2: Fundamentals of Statistics Lecture 15: Models and statistics Data from one or a series of random experiments are collected. Planning experiments and collecting data (not discussed here). Analysis:
More informationUCSD ECE250 Handout #24 Prof. Young-Han Kim Wednesday, June 6, Solutions to Exercise Set #7
UCSD ECE50 Handout #4 Prof Young-Han Kim Wednesday, June 6, 08 Solutions to Exercise Set #7 Polya s urn An urn initially has one red ball and one white ball Let X denote the name of the first ball drawn
More informationEE514A Information Theory I Fall 2013
EE514A Information Theory I Fall 2013 K. Mohan, Prof. J. Bilmes University of Washington, Seattle Department of Electrical Engineering Fall Quarter, 2013 http://j.ee.washington.edu/~bilmes/classes/ee514a_fall_2013/
More informationRandom Variables. Random variables. A numerically valued map X of an outcome ω from a sample space Ω to the real line R
In probabilistic models, a random variable is a variable whose possible values are numerical outcomes of a random phenomenon. As a function or a map, it maps from an element (or an outcome) of a sample
More informationExample continued. Math 425 Intro to Probability Lecture 37. Example continued. Example
continued : Coin tossing Math 425 Intro to Probability Lecture 37 Kenneth Harris kaharri@umich.edu Department of Mathematics University of Michigan April 8, 2009 Consider a Bernoulli trials process with
More informationFinal Examination Solutions (Total: 100 points)
Final Examination Solutions (Total: points) There are 4 problems, each problem with multiple parts, each worth 5 points. Make sure you answer all questions. Your answer should be as clear and readable
More informationMore than one variable
Chapter More than one variable.1 Bivariate discrete distributions Suppose that the r.v. s X and Y are discrete and take on the values x j and y j, j 1, respectively. Then the joint p.d.f. of X and Y, to
More informationSTAT 200C: High-dimensional Statistics
STAT 200C: High-dimensional Statistics Arash A. Amini May 30, 2018 1 / 59 Classical case: n d. Asymptotic assumption: d is fixed and n. Basic tools: LLN and CLT. High-dimensional setting: n d, e.g. n/d
More informationRecitation 2: Probability
Recitation 2: Probability Colin White, Kenny Marino January 23, 2018 Outline Facts about sets Definitions and facts about probability Random Variables and Joint Distributions Characteristics of distributions
More informationLimiting Distributions
We introduce the mode of convergence for a sequence of random variables, and discuss the convergence in probability and in distribution. The concept of convergence leads us to the two fundamental results
More information17. Convergence of Random Variables
7. Convergence of Random Variables In elementary mathematics courses (such as Calculus) one speaks of the convergence of functions: f n : R R, then lim f n = f if lim f n (x) = f(x) for all x in R. This
More informationUCSD ECE153 Handout #34 Prof. Young-Han Kim Tuesday, May 27, Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei)
UCSD ECE53 Handout #34 Prof Young-Han Kim Tuesday, May 7, 04 Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei) Linear estimator Consider a channel with the observation Y XZ, where the
More informationLecture 18: Central Limit Theorem. Lisa Yan August 6, 2018
Lecture 18: Central Limit Theorem Lisa Yan August 6, 2018 Announcements PS5 due today Pain poll PS6 out today Due next Monday 8/13 (1:30pm) (will not be accepted after Wed 8/15) Programming part: Java,
More informationLecture 11. Probability Theory: an Overveiw
Math 408 - Mathematical Statistics Lecture 11. Probability Theory: an Overveiw February 11, 2013 Konstantin Zuev (USC) Math 408, Lecture 11 February 11, 2013 1 / 24 The starting point in developing the
More informationDeep Learning for Computer Vision
Deep Learning for Computer Vision Lecture 3: Probability, Bayes Theorem, and Bayes Classification Peter Belhumeur Computer Science Columbia University Probability Should you play this game? Game: A fair
More informationProbability and Statistics
Probability and Statistics Jane Bae Stanford University hjbae@stanford.edu September 16, 2014 Jane Bae (Stanford) Probability and Statistics September 16, 2014 1 / 35 Overview 1 Probability Concepts Probability
More information1 Review of Probability
1 Review of Probability Random variables are denoted by X, Y, Z, etc. The cumulative distribution function (c.d.f.) of a random variable X is denoted by F (x) = P (X x), < x
More informationUCSD ECE250 Handout #20 Prof. Young-Han Kim Monday, February 26, Solutions to Exercise Set #7
UCSD ECE50 Handout #0 Prof. Young-Han Kim Monday, February 6, 07 Solutions to Exercise Set #7. Minimum waiting time. Let X,X,... be i.i.d. exponentially distributed random variables with parameter λ, i.e.,
More information2. Variance and Covariance: We will now derive some classic properties of variance and covariance. Assume real-valued random variables X and Y.
CS450 Final Review Problems Fall 08 Solutions or worked answers provided Problems -6 are based on the midterm review Identical problems are marked recap] Please consult previous recitations and textbook
More informationLecture 2: Convergence of Random Variables
Lecture 2: Convergence of Random Variables Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Introduction to Stochastic Processes, Fall 2013 1 / 9 Convergence of Random Variables
More informationSTAT 512 sp 2018 Summary Sheet
STAT 5 sp 08 Summary Sheet Karl B. Gregory Spring 08. Transformations of a random variable Let X be a rv with support X and let g be a function mapping X to Y with inverse mapping g (A = {x X : g(x A}
More informationStochastic Models (Lecture #4)
Stochastic Models (Lecture #4) Thomas Verdebout Université libre de Bruxelles (ULB) Today Today, our goal will be to discuss limits of sequences of rv, and to study famous limiting results. Convergence
More informationProblem Solving. Correlation and Covariance. Yi Lu. Problem Solving. Yi Lu ECE 313 2/51
Yi Lu Correlation and Covariance Yi Lu ECE 313 2/51 Definition Let X and Y be random variables with finite second moments. the correlation: E[XY ] Yi Lu ECE 313 3/51 Definition Let X and Y be random variables
More informationMidterm #1. Lecture 10: Joint Distributions and the Law of Large Numbers. Joint Distributions - Example, cont. Joint Distributions - Example
Midterm #1 Midterm 1 Lecture 10: and the Law of Large Numbers Statistics 104 Colin Rundel February 0, 01 Exam will be passed back at the end of class Exam was hard, on the whole the class did well: Mean:
More informationFundamental Tools - Probability Theory IV
Fundamental Tools - Probability Theory IV MSc Financial Mathematics The University of Warwick October 1, 2015 MSc Financial Mathematics Fundamental Tools - Probability Theory IV 1 / 14 Model-independent
More informationLecture 1: Review on Probability and Statistics
STAT 516: Stochastic Modeling of Scientific Data Autumn 2018 Instructor: Yen-Chi Chen Lecture 1: Review on Probability and Statistics These notes are partially based on those of Mathias Drton. 1.1 Motivating
More informationSTATISTICS/ECONOMETRICS PREP COURSE PROF. MASSIMO GUIDOLIN
Massimo Guidolin Massimo.Guidolin@unibocconi.it Dept. of Finance STATISTICS/ECONOMETRICS PREP COURSE PROF. MASSIMO GUIDOLIN SECOND PART, LECTURE 2: MODES OF CONVERGENCE AND POINT ESTIMATION Lecture 2:
More informationProblem Points S C O R E Total: 120
PSTAT 160 A Final Exam December 10, 2015 Name Student ID # Problem Points S C O R E 1 10 2 10 3 10 4 10 5 10 6 10 7 10 8 10 9 10 10 10 11 10 12 10 Total: 120 1. (10 points) Take a Markov chain with the
More informationLecture Notes 1 Probability and Random Variables. Conditional Probability and Independence. Functions of a Random Variable
Lecture Notes 1 Probability and Random Variables Probability Spaces Conditional Probability and Independence Random Variables Functions of a Random Variable Generation of a Random Variable Jointly Distributed
More informationProbability Review. Chao Lan
Probability Review Chao Lan Let s start with a single random variable Random Experiment A random experiment has three elements 1. sample space Ω: set of all possible outcomes e.g.,ω={1,2,3,4,5,6} 2. event
More informationLecture 11: Random Variables
EE5110: Probability Foundations for Electrical Engineers July-November 2015 Lecture 11: Random Variables Lecturer: Dr. Krishna Jagannathan Scribe: Sudharsan, Gopal, Arjun B, Debayani The study of random
More informationConvergence of Random Variables
1 / 13 Convergence of Random Variables Saravanan Vijayakumaran sarva@ee.iitb.ac.in Department of Electrical Engineering Indian Institute of Technology Bombay April 8, 2015 2 / 13 Motivation Theorem (Weak
More information3. Review of Probability and Statistics
3. Review of Probability and Statistics ECE 830, Spring 2014 Probabilistic models will be used throughout the course to represent noise, errors, and uncertainty in signal processing problems. This lecture
More informationTom Salisbury
MATH 2030 3.00MW Elementary Probability Course Notes Part V: Independence of Random Variables, Law of Large Numbers, Central Limit Theorem, Poisson distribution Geometric & Exponential distributions Tom
More informationStatistics 1B. Statistics 1B 1 (1 1)
0. Statistics 1B Statistics 1B 1 (1 1) 0. Lecture 1. Introduction and probability review Lecture 1. Introduction and probability review 2 (1 1) 1. Introduction and probability review 1.1. What is Statistics?
More informationConvergence of Random Variables
1 / 15 Convergence of Random Variables Saravanan Vijayakumaran sarva@ee.iitb.ac.in Department of Electrical Engineering Indian Institute of Technology Bombay March 19, 2014 2 / 15 Motivation Theorem (Weak
More informationProbability Theory for Machine Learning. Chris Cremer September 2015
Probability Theory for Machine Learning Chris Cremer September 2015 Outline Motivation Probability Definitions and Rules Probability Distributions MLE for Gaussian Parameter Estimation MLE and Least Squares
More information2 Statistical Estimation: Basic Concepts
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof. N. Shimkin 2 Statistical Estimation:
More informationProbability Background
Probability Background Namrata Vaswani, Iowa State University August 24, 2015 Probability recap 1: EE 322 notes Quick test of concepts: Given random variables X 1, X 2,... X n. Compute the PDF of the second
More informationRegression and Statistical Inference
Regression and Statistical Inference Walid Mnif wmnif@uwo.ca Department of Applied Mathematics The University of Western Ontario, London, Canada 1 Elements of Probability 2 Elements of Probability CDF&PDF
More information5 Operations on Multiple Random Variables
EE360 Random Signal analysis Chapter 5: Operations on Multiple Random Variables 5 Operations on Multiple Random Variables Expected value of a function of r.v. s Two r.v. s: ḡ = E[g(X, Y )] = g(x, y)f X,Y
More informationSection 9.1. Expected Values of Sums
Section 9.1 Expected Values of Sums Theorem 9.1 For any set of random variables X 1,..., X n, the sum W n = X 1 + + X n has expected value E [W n ] = E [X 1 ] + E [X 2 ] + + E [X n ]. Proof: Theorem 9.1
More informationECE534, Spring 2018: Solutions for Problem Set #3
ECE534, Spring 08: Solutions for Problem Set #3 Jointly Gaussian Random Variables and MMSE Estimation Suppose that X, Y are jointly Gaussian random variables with µ X = µ Y = 0 and σ X = σ Y = Let their
More informationChapter 6: Large Random Samples Sections
Chapter 6: Large Random Samples Sections 6.1: Introduction 6.2: The Law of Large Numbers Skip p. 356-358 Skip p. 366-368 Skip 6.4: The correction for continuity Remember: The Midterm is October 25th in
More informationProbability and Measure
Chapter 4 Probability and Measure 4.1 Introduction In this chapter we will examine probability theory from the measure theoretic perspective. The realisation that measure theory is the foundation of probability
More informationLecture 2 Sep 5, 2017
CS 388R: Randomized Algorithms Fall 2017 Lecture 2 Sep 5, 2017 Prof. Eric Price Scribe: V. Orestis Papadigenopoulos and Patrick Rall NOTE: THESE NOTES HAVE NOT BEEN EDITED OR CHECKED FOR CORRECTNESS 1
More informationChapter 3, 4 Random Variables ENCS Probability and Stochastic Processes. Concordia University
Chapter 3, 4 Random Variables ENCS6161 - Probability and Stochastic Processes Concordia University ENCS6161 p.1/47 The Notion of a Random Variable A random variable X is a function that assigns a real
More informationLecture 4: Law of Large Number and Central Limit Theorem
ECE 645: Estimation Theory Sring 2015 Instructor: Prof. Stanley H. Chan Lecture 4: Law of Large Number and Central Limit Theorem (LaTeX reared by Jing Li) March 31, 2015 This lecture note is based on ECE
More informationLecture 22: Variance and Covariance
EE5110 : Probability Foundations for Electrical Engineers July-November 2015 Lecture 22: Variance and Covariance Lecturer: Dr. Krishna Jagannathan Scribes: R.Ravi Kiran In this lecture we will introduce
More informationChapter 6. Convergence. Probability Theory. Four different convergence concepts. Four different convergence concepts. Convergence in probability
Probability Theory Chapter 6 Convergence Four different convergence concepts Let X 1, X 2, be a sequence of (usually dependent) random variables Definition 1.1. X n converges almost surely (a.s.), or with
More informationStationary independent increments. 1. Random changes of the form X t+h X t fixed h > 0 are called increments of the process.
Stationary independent increments 1. Random changes of the form X t+h X t fixed h > 0 are called increments of the process. 2. If each set of increments, corresponding to non-overlapping collection of
More informationIEOR E4703: Monte-Carlo Simulation
IEOR E4703: Monte-Carlo Simulation Output Analysis for Monte-Carlo Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com Output Analysis
More informationLecture Notes 1 Probability and Random Variables. Conditional Probability and Independence. Functions of a Random Variable
Lecture Notes 1 Probability and Random Variables Probability Spaces Conditional Probability and Independence Random Variables Functions of a Random Variable Generation of a Random Variable Jointly Distributed
More information1: PROBABILITY REVIEW
1: PROBABILITY REVIEW Marek Rutkowski School of Mathematics and Statistics University of Sydney Semester 2, 2016 M. Rutkowski (USydney) Slides 1: Probability Review 1 / 56 Outline We will review the following
More information2 n k In particular, using Stirling formula, we can calculate the asymptotic of obtaining heads exactly half of the time:
Chapter 1 Random Variables 1.1 Elementary Examples We will start with elementary and intuitive examples of probability. The most well-known example is that of a fair coin: if flipped, the probability of
More informationCOMP2610/COMP Information Theory
COMP2610/COMP6261 - Information Theory Lecture 9: Probabilistic Inequalities Mark Reid and Aditya Menon Research School of Computer Science The Australian National University August 19th, 2014 Mark Reid
More informationVariance reduction. Michel Bierlaire. Transport and Mobility Laboratory. Variance reduction p. 1/18
Variance reduction p. 1/18 Variance reduction Michel Bierlaire michel.bierlaire@epfl.ch Transport and Mobility Laboratory Variance reduction p. 2/18 Example Use simulation to compute I = 1 0 e x dx We
More informationLecture 9: March 26, 2014
COMS 6998-3: Sub-Linear Algorithms in Learning and Testing Lecturer: Rocco Servedio Lecture 9: March 26, 204 Spring 204 Scriber: Keith Nichols Overview. Last Time Finished analysis of O ( n ɛ ) -query
More informationEvaluating Classifiers. Lecture 2 Instructor: Max Welling
Evaluating Classifiers Lecture 2 Instructor: Max Welling Evaluation of Results How do you report classification error? How certain are you about the error you claim? How do you compare two algorithms?
More informationLecture Notes 3 Multiple Random Variables. Joint, Marginal, and Conditional pmfs. Bayes Rule and Independence for pmfs
Lecture Notes 3 Multiple Random Variables Joint, Marginal, and Conditional pmfs Bayes Rule and Independence for pmfs Joint, Marginal, and Conditional pdfs Bayes Rule and Independence for pdfs Functions
More informationLecture 4 : Random variable and expectation
Lecture 4 : Random variable and expectation Study Objectives: to learn the concept of 1. Random variable (rv), including discrete rv and continuous rv; and the distribution functions (pmf, pdf and cdf).
More information6.041/6.431 Fall 2010 Quiz 2 Solutions
6.04/6.43: Probabilistic Systems Analysis (Fall 200) 6.04/6.43 Fall 200 Quiz 2 Solutions Problem. (80 points) In this problem: (i) X is a (continuous) uniform random variable on [0, 4]. (ii) Y is an exponential
More informationExpectation. DS GA 1002 Probability and Statistics for Data Science. Carlos Fernandez-Granda
Expectation DS GA 1002 Probability and Statistics for Data Science http://www.cims.nyu.edu/~cfgranda/pages/dsga1002_fall17 Carlos Fernandez-Granda Aim Describe random variables with a few numbers: mean,
More informationLecture Notes 2 Random Variables. Discrete Random Variables: Probability mass function (pmf)
Lecture Notes 2 Random Variables Definition Discrete Random Variables: Probability mass function (pmf) Continuous Random Variables: Probability density function (pdf) Mean and Variance Cumulative Distribution
More informationECE 450 Homework #3. 1. Given the joint density function f XY (x,y) = 0.5 1<x<2, 2<y< <x<4, 2<y<3 0 else
ECE 450 Homework #3 0. Consider the random variables X and Y, whose values are a function of the number showing when a single die is tossed, as show below: Exp. Outcome 1 3 4 5 6 X 3 3 4 4 Y 0 1 3 4 5
More informationTHEORY OF PROBABILITY VLADIMIR KOBZAR
THEORY OF PROBABILITY VLADIMIR KOBZAR Lecture 20 - Conditional Expectation, Inequalities, Laws of Large Numbers, Central Limit Theorem This lecture is based on the materials from the Courant Institute
More informationLecture 2: Review of Probability
Lecture 2: Review of Probability Zheng Tian Contents 1 Random Variables and Probability Distributions 2 1.1 Defining probabilities and random variables..................... 2 1.2 Probability distributions................................
More information