General Random Variables

Size: px

Start display at page:

Download "General Random Variables"

Mary Walton
5 years ago
Views:

1 1/65 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Probability

2 A general random variable is discrete, continuous, or mixed. A discrete random variable has a countable image (of a sample space). A continuous random variable has an uncountable image which is a continuous set of numbers. A mixed random variable has both, 2/65

3 Continuous Random Variables A continuous random variable arises from an uncountable sample space, e.g. when the underlying random experiment involves a measurement such as time position the velocity of a vehicle on highway 3/65

4 Point vs. Neighborhood The way to see a continuous random variable is to consider a small neighborhood, rather than to look at an isolated point. The event of a continuous random variable X taking a value in the neighborhood of x of length δ will be denoted by X (x, δ) 4/65

5 Probability of an Infinitesimal Event Random variable X is a continuous. Consider P(X (x, δ)), the probability of event X (x, δ). P(X (x, δ)) depends on x. P(X (x, δ)) is proportional to δ. Such a probability is specified by a probability density function (PDF) f X evaluated at x X (x, δ) P(X (x, δ)) = f X (x)δ 5/65

6 Probability of an Event Random variable X is continuous and defined on a probability model (Ω, F, P), with image I X and PDF f X. Then the probability of event X B, where B is a countable union of intervals in I X, is an integration P(X B) = f X (x)dx B X (x, δ) P(X (x, δ)) = f X (x)δ B = (x i, δ i ) X B P(X B) = f X (x)dx B 6/65

7 Probability Based on a Continuous RV Refer to Figure 3.1 and Figure 3.2. A probability model based on a continuous random variable X is completely specified by image I X and PDF f X. We represent such a model by (X, I X, f X ) 7/65

8 Discrete RV vs. Continuous RV For a discrete random variable X of (X, I X, g X ), the image I X is countable, and g X = p X is a probability mass function which is used to assign probabilities to atomic events X = x i s. For a continuous random variable X of (X, I X, g X ), the image I X is uncountable, and g X = f X is a probability density function for assigning infinitesimal probabilities to infinitesimal events, which are extremely-small intervals of non-zero lengths. Infinitesimal events constitute a countable partition of Ω. An event is a countable union of infinitesimal events. The probability of an event is an integration of infinitesimal probabilities. 8/65

9 Properties of PDF totality fx dx = 1 Random variable X has PDF f X. Then we require non-negativity f X 0 9/65

10 Continuous Uniform Random Variable Random variable X of (X, I X, f X ) is uniform if f X is constant over I X. A continuous uniform random variable X with image I X = [a, b] is denoted by X uniform(a, b) The PDF of X is f X (x) = { 1 b a, a x b 0, otherwise 10/65

11 Example 3.1 A wheel of fortune is fair and continuously calibrated between 0 and 1. What is the PDF of the result of a spin of the wheel? Assuming that the wheel is fair, we can model this experiment in terms of a continuous uniform random variable X with PDF { c, 0 x 1 f X (x) = 0, otherwise The constant c is decided by f X (x)dx = c dx = 1 c = 1 11/65

12 Example 3.2 Alvin s driving time to work is between 15 and 20 minutes in a sunny day, and between 20 and 25 minutes in a rainy day, with all times being equally likely in each case. Assume that a day is sunny with probability 2/3, and rainy with probability 1/3. What is the PDF of the driving time, viewed as a random variable X? PDF f X is piece-wise constant = c 1 dx, = c 2 dx 20 c 1 = 2 15, c 2 = /65

13 Mean, Variance, Moments, etc. E[X] = xf X dx var(x) = E[(X E[X]) 2 ] = E[X 2 ] (E[X]) 2 ( ) 2 = x 2 f X dx xf X dx E[X n ] = x n f X dx E[g(X)] = g(x)f X (x)dx E[aX + b] = ae[x] + b, var(ax + b) = a 2 var(x) 13/65

14 Example 3.4 Random variable X is continuous with X uniform(a, b) What are the mean and variance of X? 14/65

15 Exponential Random Variable An exponential random variable T with parameter α > 0 has image I T = {t 0} and PDF This is denoted by f T (t) = { αe αt, t 0 0, t < 0 T exponential(α) See Figure 3.5 for examples of exponential PDFs. 15/65

16 Mean and Variance Random variable T is continuous with T exponential(α). So E[T ] = t f T (t)dt = = ( te αt ) 0 E[T 2 ] = t 2 f T (t)dt = = t2 e αt + 1 α α 0 0 t ( αe αt) dt + e αt dt = 1 α t 2 αe αt dt var(t ) = E[T 2 ] E 2 [T ] = 1 α 2 2te αt dt = 2 α 2 16/65

17 Example 3.5 Meteorite The time until a small meteorite first lands anywhere in the Sahara desert is modeled as an exponential random variable with a mean of 10 days. The time is currently midnight. What is the probability that a meteorite first lands sometime between 6 a.m. and 6 p.m. of the first day? T exponential(α), E[T ] = 10 = 1 α α = 1 10 ( 1 P 4 T = f 4) 1 T (t)dt = 4 = e αt = e 1 40 e αe αt dt 17/65

18 Cumulative Distribution Functions 18/65

19 Definition By definition, the cumulative distribution function (CDF) of a random variable X is F X (x) = P(X x) If X is discrete If X is continuous F X (x) = p X (x i ) x i I X,x i x x F X (x) = fx (t)dt See Figure 3.6 and Figure /65

20 Properties of CDF non-decreasing x 1 x 2 F X (x 1 ) F X (x 2 ) limits CDF and PDF lim F X (x) = 0, x lim F X (x) = 1 x CDF and PMF x F X (x) = fx (t)dt, f X = df X dx F X (x) = x i x p X (x i ), p X (x k ) = F X (x k ) F X (x k 1 ) 20/65

21 Example 3.6 Take 3 tests, and the final score is the maximum of the test scores. Assume that the score in each test is from 1 to 10, each with probability 1/10, independent of the scores in other tests. What is the PMF of the final score X? Random variable X i is the score for test i. Then X = max(x 1, X 2, X 3 ) (X x) = (X 1 x) (X 2 x) (X 3 x) ( x 3 F X (x) = F X1 (x)f X2 (x)f X3 (x) = 10) ( ) k 3 ( k 1 p X (k) = F X (k) F X (k 1) = ) 3 21/65

22 CDF of a Geometric Random Variable Random variable X is discrete with X geometric(p). The CDF of X is x F X (x) = p X (k) k=1 = 1 (1 p) x, x 0 At the positive integers F X (n) = 1 (1 p) n 22/65

23 CDF of an Exponential Random Variable Random variable Y is continuous with Y exponential(α). The CDF of Y is y F Y (y) = fy (t)dt = y 0 αe αt dt = 1 e αy, y 0 At the positive integers F Y (n) = 1 (e α ) n 23/65

24 Exponential and Geometric An exponential random variable, which is continuous, can be approximated by a geometric random variable, which is discrete. Random variable Y exponential(α) and X geometric(p). If we set p in X to have then 1 p = e α or equivalently p = 1 e α F Y (n) = F X (n) at the positive integers. The agreement between F Y and F X only at the positive integers is somewhat coarse. 24/65

25 Finer Approximation Refer to Figure 3.8. For a finer approximation, we introduce a granularity parameter δ and ask CDFs to agree at every δ. Specifically, we define discrete random variable W = δ X and require F Y (nδ) = F W (nδ), n = 1, 2,... Since (W nδ) = (X n), this is equivalent to Thus, we set p to F Y (nδ) = F W (nδ) = F X (n) 1 e αnδ = 1 (1 p) n 1 p = e αδ p = 1 e αδ In summary, Y exponential(α) is approximated by W = δx. W is the time of the first success in a sequence of independent Bernoulli trials conducted every δ. X is Bernoulli with parameter p = 1 e αδ 25/65

26 Normal Random Variables (Gaussians) 26/65

27 Parameters in PDF A normal random variable X with parameters µ and σ 2, denoted by X N (µ, σ 2 ) has image I X = R and PDF f X = 1 e (x µ)2 2σ 2 2πσ a single peak at µ large σ 2, short and fat PDF see Figure /65

28 Properties of Normal Random Variables Random variable X is normal, with X N (µ, σ 2 ). f X dx = 1 E[X] = µ var(x) = σ 2 Good exercise of calculus to prove these results. 28/65

29 Standard Normal Random variable Y is a standard normal random variable if Y is normal with zero mean and unit variance. That is Y N (0, 1) The CDF of Y is specifically denoted by Φ(y) Φ(y) = P(Y y) = y 1 2π e t2 /2 dt The values of Φ(y) for y > 0 are listed in a table. For y < 0 Φ(y) = P(Y y) = 1 P(Y y) = 1 Φ( y) 29/65

30 Normal and Standard Normal Easy conversion between a normal RV and a standard normal RV. Random variable Y is standard normal. Then X = σy + µ is a normal random variable with mean µ and variance σ 2. Random variable X is normal with mean µ and variance σ 2. Then Y = X µ σ is a standard normal random variable. 30/65

31 CDF of a Normal Random Variable By a simple linear transformation, the CDF of a normal RV is related to the CDF of a standard normal RV. Random variable X is normal with X N (µ, σ 2 ). Then ( X µ P(X x) = P x µ ) σ σ ( = P Y x µ ) σ ( ) x µ = Φ σ where random variable Y is standard normal. 31/65

32 Example 3.7 The yearly snowfall at Mountain Rainier is modeled as a normal random variable with a mean of µ = 60 and a standard deviation of σ = 20. What is the probability that this year s snowfall will be at least 80 inches? Random variable X is the snowfall this year. Then P(X 80) = 1 P(X 80) ( X 60 = 1 P 20 = 1 P(Y 1) = 1 Φ(1) = = ) /65

33 Example 3.8 Signal Detection A binary message is transmitted as a signal S, which is either 1 or +1. The channel corrupts the transmission with an additive normal noise N with mean 0 and variance σ 2. The receiver receives Y = S + N and decides that S = 1 (or S = +1) if Y < 0 (or Y 0). What is the probability of error event E? P(E) = P(E (S = 1)) + P(E (S = 1)) = P(E S = 1)P(S = 1) + P(E S = 1)P(S = 1) = P(Y < 0 S = 1)P(S = 1) + P(Y > 0 S = 1)P(S = 1) = P(N 1)P(S = 1) + P(N 1)P(S = 1) ( N 0 = P(N 1) = 1 P(N < 1) = 1 P < 1 0 ) σ σ ( ) 1 = 1 Φ σ 33/65

34 Multiple Continuous Random Variables 34/65

35 Joint Probability Density Function Random variables X and Y are continuous and defined on a model (Ω, F, P). We assign probability to infinitesimal event X (x, δ x ) Y (y, δ y ) The assignment is through a joint probability density function (joint PDF) f XY P(X (x, δ x ) Y (y, δ y )) = f XY (x, y)δ x δ y The probability model of X and Y is completely specified by image I XY and a joint PDF f XY, so it can be represented by ((X, Y ), I XY, f XY ) 35/65

36 The Probability of an Event Random variables X and Y are continuous. The probability of the event (X, Y ) B is a double integration of the joint PDF of X and Y P((X, Y ) B) = f XY dxdy B 36/65

37 Properties of Joint PDF Random variables X and Y are continuous. The joint PDF of X and Y must satisfy non-negativity totality f XY 0 f XY dxdy = 1 37/65

38 Marginalization Random variables X and Y are continuous. The joint PDF f XY contains complete information of the probability of X and Y. Marginal PDFs f X and f Y can be derived from joint PDF f XY. The probability of the event X A is Similarly P(X A) = P((X A) (Y (, ))) A f X dx = A f XY dydx f X = f XY dy f Y = f XY dx 38/65

39 Example 3.9 Uniform PDF Random variables X and Y are the arrival times of Romeo and Juliet. Assume them to be continuous and uniform. What is the joint PDF? Since there is no preference to any time, the joint PDF is constant { c, if 0 x 1 and 0 y 1 f XY (x, y) = 0, otherwise c is determined by f XY (x, y)dxdy = 1 c = 1 39/65

40 Example 3.10 Random variables X and Y are continuous. The joint PDF of X and Y is a constant c on S, as shown in Figure 3.12, and is zero outside S. Determine the value of c and f X. { c, if (x, y) S f XY (x, y) = 0, otherwise f XY dxdy = 1 c = 1 4 f X = 3 4, if 1 x 2 f XY dy = 1 4, if 2 x 3 0, otherwise 40/65

41 Example 3.11 Buffon s Needle A surface is ruled with parallel lines, which are at distance d from each other. A needle of length l is thrown on the surface at random. Assume that l < d. What is the probability that the needle will intersect one of the lines? 41/65

42 Refer to Figure Random variable X is the distance from the center of needle to the nearest line, and Θ is the acute angle between needle and line. The joint PDF f XΘ is uniform f XΘ = f X f Θ = 4 πd, 0 x d 2, 0 θ π 2 P(needle crossing line) = P (X l ) 2 sin Θ = f XΘ dxdθ x l 2 sin θ = π 2 0 = 2l πd = 2l πd l 2 sin θ 0 π 2 0 sin θdθ 4 πd dxdθ 42/65

43 Conditional Probability 43/65

44 Conditioning on an Event Random variable X is continuous and event A is non-null. The conditional probability of event X (x, δ) given A is proportional to δ and depends on x, so it can be written as P(X (x, δ) A) = f X A (x)δ We call f X A the conditional probability density function (conditional PDF) of X given A. The conditional probability of X B given A is an integration of conditional PDF P(X B A) = f X A (x)dx B 44/65

45 Conditioning on X C Conditioning on event X C, we have P((X (x, δ)) (X C)) P(X (x, δ) X C) = P(X C) f X (x)δ = ((x, δ) C)? P(X C) : 0 = f X {X C} (x)δ f X {X C} (x) = ((x, δ) C)? 1 That is, f X {X C} (x) either magnifies by depending on whether x C. Figure f X (x) P(X C) : 0 P(X C) or vanishes, 45/65

46 Example 3.13 A Memoryless Random Variable The time T until a new light bulb burns out is an exponential random variable with parameter α. Alice turns the light on, leaves the room, and when she returns, t time units later, finds that the light bulb is still on, which corresponds to event A = (T > t). X is the additional time until the light bulb eventually burns out. What is the conditional CDF of X given A? P(X x A) = P(T t + x T > t) = = P((T t + x) (T > t)) P(T > t) P(t < T t + x) P(T > t) = e αt e α(t+x) e αt Note that P(X x T > t) is independent of t. = 1 e αx 46/65

47 Total Probability Theorem Random variable X is continuous and defined on (Ω, F, P), and {A 1,..., A n } is a partition of Ω. Then n f X (x) = P(A i )f X Ai (x) i=1 n F X (x) = P(X x) = P(X x A i ) i=1 n n x = P(A i )P(X x A i ) = P(A i ) fx Ai (x )dx i=1 i=1 x n x = P(A i )f X Ai (x )dx = fx (x )dx i=1 n f X (x) = P(A i )f X Ai (x) i=1 47/65

48 Example 3.14 Waiting Time A train arrives at a station every quarter hour starting at 6:00am. You walk into the station between 7:10am and 7:30am, and your arrival time is uniform over this interval. What is the PDF of the time you have to wait for the first train to arrive? Event A = {catch the 7:15 train} and random variable X is the waiting time. P(A) = 1 4, f X A(x) = 1 5, 0 x 5 P(A c ) = 3 4, f X A c (x) = 1 15, 0 x 15 By the total probability theorem f X (x) = P(A)f X A (x) + P(A c )f X A c (x) { 1 = , 0 x , 5 x 15 48/65

49 Conditioning on Continuous Random Variable Random variables X and Y are continuous and defined on (Ω, F, P). The conditional probability of event X (x, δ x ) given event Y (y, δ y ) is P(X (x, δ x ) Y (y, δ y )) = P((X (x, δ x)) (Y (y, δ y ))) P(Y (y, δ y )) = f XY (x, y)δ x δ y = f XY (x, y) δ x f Y (y)δ y f Y (y) It does not depend on δ y and it is proportional to δ x, so we write P(X (x, δ x ) Y (y, δ y )) = P(X (x, δ x ) Y = y) = f X Y (x y)δ x where f X Y (x y), called conditional probability density function (conditional PDF), specifies the probability density of X at x when Y = y. 49/65

50 Chain Rule Equating the expressions of P(X (x, δ x ) Y (y, δ y )), we get f X Y = f XY f Y We have shown that marginal PDF f Y can be derived from joint PDF f XY. Thus, conditional PDF f X Y can also be derived from joint PDF f XY. Conversely, since f XY = f X Y f Y we can derive f XY given f Y and f X Y (or given f X and f Y X ). 50/65

51 Example 3.15 Darting Ben throws a dart at a circular target of radius r. The random point of impact is (X, Y ). We assume that he always hits the target, and that all points of impact are equally likely. What is the conditional PDF f X Y? f Y = f XY dx = f X Y = f XY f Y = f X Y = f XY f Y r 2 y 2 1 r 2 y 2 πr 2 dx = 2 πr 2 r 2 y r 2 y, x r 2 y /65

52 Example 3.16 Police Radar The speed of a vehicle that drives past a police radar is modeled as an exponential random variable X with mean 50 miles per hour. The police radar s measurement Y of the vehicle s speed has an error which is modeled as a normal random variable with zero mean and standard deviation equal to one tenth of the vehicle s speed. What is the joint PDF of X and Y? f XY = f X f Y X = αe αx 1 2πσY X (y µ Y X )2 2σ e 2 Y X = 1 50 e x 50 1 (y x)2 ( 2π x e 2( 10) 10) x 2 52/65

53 Conditional Expectation Random variable X is continuous and defined on (Ω, F, P). 1 conditioning on an event: A is a non-null event. The conditional expectation of X given A is E[X A] = x f X A (x)dx 2 conditioning on a random variable: Y is continuous and is defined on (Ω, F, P). The conditional expectation of X given Y = y is E[X Y = y] = x f X Y (x y)dx 53/65

54 Total Expectation Random variables X and Y are continuous. Then E[X] = E[X Y = y] f Y (y)dy E[X] = xf X (x)dx = x f XY (x, y)dydx = x f X Y (x y)f Y (y)dydx ( ) = xf X Y (x y)dx f Y (y)dy = E[X Y = y] f Y (y)dy 54/65

55 Total Expectation Theorem Random variable X is continuous and defined on (Ω, F, P), and {A 1,..., A n } is a partition of Ω with P(A i ) > 0. Then n E[X] = P(A i )E[X A i ] i=1 E[X] = xf X (x)dx n = x P(A i )f X Ai (x)dx i=1 n = P(A i ) xf X Ai (x)dx i=1 n = P(A i )E[X A i ] i=1 55/65

56 Example 3.17 Random variable X is continuous with image I X = [0, 2] and PDF 1/3, 0 x 1 f X (x) = 2/3, 1 < x 2 0, otherwise Find E[X] and var(x) via total expectation with the partition {A 1 = (X [0, 1]), A 2 = (X [1, 2])} 56/65

57 Independent Random Variables Random variables X and Y are continuous. They are independent if f XY (x, y) = f X (x)f Y (y) This is denoted by X Y Random variables X Y. For any y I Y f X Y (x y) = f X (x) For any x I X f Y X (y x) = f Y (y) 57/65

58 Example 3.18 Independent Gaussians Random variables Xand Y are normal with means µ x, µ y and variances σ 2 x, σ 2 y, respectively, with X Y. What is the joint PDF of X and Y? 1 f XY = f X f Y = e (x µx ) 2σx 2 2πσx 2 1 e (y µy ) 2σy 2 2πσy 2 58/65

59 Bayes Rule 59/65

60 Inference Problem Random variable X has PDF f X and Y has conditional PDF f Y X. The decision of f X Y is called inference problem. The estimation of X given Y is a decoding problem. A common decoding criterion is ˆx = arg max f X Y (x y) The decoding-error event is E = (X ˆX) 60/65

61 Two Continuous Random Variables Random variables X and Y are continuous. Given f X and f Y X, the conditional PDF of X given Y is f X Y (x y) = f X (x)f Y X (y x) fx (x )f Y X (y x )dx Joint PDF f XY (x, y) = f X (x)f Y X (y x) Marginal PDF f Y (y) = f XY (x, y)dx = f X (x )f Y X (y x )dx Conditional PDF f X Y (x y) = f XY (x, y) f Y (y) = f X (x)f Y X (y x) fx (x )f Y X (y x )dx 61/65

62 Example 3.19 Light Bulb A light bulb is known to have an exponentially distributed lifetime Y. However, the manufacturing company is experiencing quality control problems, so the parameter Λ of the PDF of Y is random, uniformly distributed in the interval [1, 3/2]. We test a light bulb and record its lifetime y. What can we say about the parameter λ? Λ and Y are continuous. The joint PDF is f ΛY (λ, y) = f Λ (λ)f Y Λ (y λ) By Bayes rule f Λ Y (λ y) = f ΛY (λ, y) = f Y (y) = ( 1 λ 3 ) 2 f Λ (λ)f Y Λ (y λ) fλ (λ )f Y Λ (y λ )dλ? 2λe λy : λ e λ y dλ 62/65

63 A Discrete RV and a Continuous RV Random variable N is discrete and Y is continuous, both defined on (Ω, F, P). The probability of a joint event is P(N = n Y (y, δ)) = P(N = n)p(y (y, δ) N = n) = p N (n)f Y N (y n)δ The conditional PMF of N given Y (y, δ) is P(N = n Y (y, δ)) p N Y (n y) = P(N = n Y (y, δ)) = P(Y (y, δ)) P(N = n Y (y, δ)) = n P(N = n Y (y, δ)) p N (n)f Y N (y n)δ = n p N(n )f Y N (y n )δ p N (n)f Y N (y n) = n p N(n )f Y N (y n ) 63/65

64 Example 3.20 Binary Signal Transmission The simplest case is a binary RV. A signal S is transmitted, with P(S = 1) = p and P(S = 1) = 1 p. The received signal is Y = S + N, where N is a normal noise with zero mean and unit variance. What is the probability that S = 1, as a function of the observed value y of Y? p S (1)f Y S (y 1) p S Y (1 y) = p S (1)f Y S (y 1) + p S ( 1)f Y S (y 1) = = p 1 2π e (y 1)2 /2 p 1 2π e (y 1)2 /2 + (1 p) 1 2π e (y+1)2 /2 pe y pe y + (1 p)e y 64/65

65 Inference Based on Event Observation Random variable Y is continuous and event A is non-null. Given f Y (y) and P(A Y = y), the conditional PDF f Y A (y) is f Y A (y) = f Y (y)p(a Y = y) P(A) f Y (y)p(a Y = y) = fy (y )P(A Y = y )dy 65/65

1 Random Variable: Topics

Note: Handouts DO NOT replace the book. In most cases, they only provide a guideline on topics and an intuitive feel. 1 Random Variable: Topics Chap 2, 2.1-2.4 and Chap 3, 3.1-3.3 What is a random variable?