Chapter 3: Random Variables 1

Transcription

1 Chapter 3: Random Variables 1 Yunghsiang S. Han Graduate Institute of Communication Engineering, National Taipei University Taiwan yshan@mail.ntpu.edu.tw 1 Modified from the lecture notes by Prof. Mao-Ching Chiu

2 Y. S. Han Random Variables The Notion of a Random Variable A random variable X is a function that assigns a real number, X(ζ), to each outcome ζ in the sample space.

3 Y. S. Han Random Variables 2 Example: Toss a coin three times. S = {HHH, HHT, HTH, THH, HTT, THT, TTH, TTT}. Let X be the number of heads. ζ HHH HHT HTH THH HTT THT TTH TTT X(ζ) X is a random variable with S X = {0, 1, 2, 3}.

4 Y. S. Han Random Variables 3 S: Sample space of a random experiment X: Random variable X : S S X S X is a new sample space Let B S X and A = {ζ : X(ζ) B}. Then P[B] = P[A] = P[{ζ : X(ζ) B}] A and B are equivalent events.

5 Y. S. Han Random Variables 4

6 Y. S. Han Random Variables Cumulative Distribution Function Cumulative distribution function (cdf) F X (x) = P[X x] for < x < In underlying sample space F X (x) = P[{ζ : X(ζ) x}] F X (x) is a function of the variable x.

7 Y. S. Han Random Variables 6 Properties of cdf 1. 0 F X (x) lim x F X (x) = lim x F X (x) = If a < b, then F X (a) F X (b). 5. F X (x) is continuous from the right, i.e., for h > 0 F X (b) = lim h 0 F X (b + h) = F X (b + ). 6. P[a < X b] = F X (b) F X (a), since {X a} {a < X b} = {X b}.

8 Y. S. Han Random Variables 7 7. P[X = b] = F X (b) F X (b ). 8. P[X > x] = 1 F X (x).

9 Y. S. Han Random Variables 8 X: the number of heads in three tosses of a fair coin Let δ be a small positive number. Then F X (1 δ) = P[X 1 δ] = P {0 heads} = 1/8, F X (1) = P[X 1] = P[0 or 1 heads] = 1/8+3/8 = 1/2, F X (1 + δ) = P[X 1 + δ] = P[0 or 1 heads] = 1/2. Write in unit step function u(x) = { 0, x < 0 1, x 0, F X (x) = 1 8 u(x) u(x 1) u(x 2) + 1 u(x 3). 8

10 Y. S. Han Random Variables 9

11 Y. S. Han Random Variables 10 Example: The transmission time X of messages in a communication system obeys the exponential probability law with parameter λ, i.e., P[X > x] = e λx x > 0 Find the cdf of X and P[T < X 2T], where T = 1/λ. F X (x) = P[X x] = 1 P[X > x] = { 0, x < 0 1 e λx, x 0. P[T < X 2T] = 1 e 2 (1 e 1 ) = e 1 e 2.

12 Y. S. Han Random Variables 11

13 Y. S. Han Random Variables 12 Types of random variables Discrete random variable, S X = x 0,x 1,... F X (x) = k p X (x k )u(x x k ), where x k S X and p X (x k ) = P[X = x k ] is the probability mass function (pmf) of X. Continuous random variable F X (x) = x Random variable of mixed type f(t)dt F X (x) = pf 1 (x) + (1 p)f 2 (x)

14 Y. S. Han Random Variables Probability Density Function Probability density function (pdf) of X, if it exists, is f X (x) = df X(x). dx

15 Y. S. Han Random Variables 14

16 Y. S. Han Random Variables 15 Properties of pdf 1. f X (x) 0 due to nondecreasing property of cdf. 2. P[a X b] = b a f X(x)dx. 3. F X (x) = x f X(t)dt. 4. f X(t)dt = 1.

17 Y. S. Han Random Variables 16 The pdf of the uniform random variable f X (x) = F X (x) = { 1 b a, a x b 0, otherwise 0, x < a x a, a x b b a 1, x > b

18 Y. S. Han Random Variables 17

19 Y. S. Han Random Variables 18 Pdf for discontinuous cdf Unit step function u(x) = { 0 x < 0 1 x 0 Delta function δ(t) u(x) = x δ(t)dt Cdf for a discrete random variable F X (x) = k p X (x k )u(x x k ) = x f X (t)dt

20 Y. S. Han Random Variables 19 f X (x) = k p X(x k )δ(x x k )

21 Y. S. Han Random Variables 20 Conditional cdf s and pdf s Conditional cdf of X given A F X (x A) = P[{X x} A] P[A] if P[A] > 0 Conditional pdf of X given A f X (x A) = d dx F X(x A)

22 Y. S. Han Random Variables Some Important Random Variables Bernoulli random variable: Let A be an event. The indicator function for A is { 0 ζ / A I A (ζ) = 1 ζ A. I A is the Bernoulli random variable. Ex: toss a coin. Binomial random variable: Let X be the number of times a event A occurs in n independent trials. Let I j be the indicator function for event A in the jth trial. Then X = I 1 + I I n

23 Y. S. Han Random Variables 22 and P[X = k] = ( n k) p k (1 p) n k, where p = P[I j = 1].

24 Y. S. Han Random Variables 23 Geometric random variable: Count the number M of independent Bernoulli trials until the first success of event A. P[M = k] = (1 p) k 1 p k = 1, 2,..., where p = P[A]. Another version of the geometric random variable is P[k] = (1 p) k p k = 0, 1, 2,....

25 Y. S. Han Random Variables 24 Exponential random variable f X (x) = { 0 x < 0 λe λx x 0 F X (x) = { 0 x < 0 1 e λx x 0 λ: rate at which events occur.

26 Y. S. Han Random Variables 25

27 Y. S. Han Random Variables 26 The Poisson random variable: The pmf is P[N = k] = αk k! e α k = 0, 1, 2,,..., where α is the average number of event occurrences in a specified time interval or region in space. The pmf of the Poison random variable sums to one, since k=0 α k k! e α = e α k=0 α k k! = e α e α = 1.

28 Y. S. Han Random Variables 27

29 Y. S. Han Random Variables 28 Gaussian (Normal) random variable: The pdf is f X (x) = 1 2πσ e (x m)2 /2σ 2 < x <, where m and σ are real numbers. The cdf is P[X x] = 1 2πσ x e (x m) 2 /2σ 2 dx. Change variable t = (x m)/σ and we have F X (x) = 1 (x m)/σ ( ) x m e t2 /2 dt = Φ 2π σ,

30 Y. S. Han Random Variables 29 where Φ(x) = 1 2π x e t2 /2 dt

31 Y. S. Han Random Variables 30

32 Y. S. Han Random Variables 31 Q-function is defined by Q(x) = 1 Φ(x) = 1 2π x e t2 /2 dt. Q-function is the probability of tail of the pdf. Q(0) = 1/2 and Q( x) = 1 Q(x). Q(x) can be obtained by look-up tables.

33 Y. S. Han Random Variables 32 Example: A communication system accepts a positive voltage V as input and output a voltage Y = αv + N, where α = 10 2 and N is a Gaussian random variable with parameters m = 0 and σ = 2. Find the value of V that gives P[Y < 0] = Sol: P[Y < 0] = P[αV + N < 0] = P[N < αv ] ( ) ( ) αv αv = Φ = Q = σ σ From the Q-function table, we have αv/σ = Thus, V = (4.753)σ/α =

34 Y. S. Han Random Variables Functions of a Random Variable Let X be a random variable. Define another random variable Y = g(x). Example: Let the function h(x) = (x) + be defined as (x) + = { 0 x < 0 x x 0. Let B = {x : g(x) C}. The probability of event C is P[Y C] = P[g(X) C] = P[X B]. Three types of equivalent events are useful in determining the cdf and pdf:

35 Y. S. Han Random Variables Discontinuity case: {g(x) = y k }; 2. cdf: {g(x) y}; 3. pdf: {y < g(x) y + h}.

36 Y. S. Han Random Variables 35 Example: Let X be a sample voltage of a speech waveform, and suppose that X has a uniform distribution in the interval [ 4d, 4d]. Let Y = q(x), where the quantizer input-output characteristic is shown below. Find the pmf for Y.

37 Y. S. Han Random Variables 36 Sol: The event {Y = q} for q in S Y is equivalent to the event {X I q }, where I q is an interval of points mapped into the

38 Y. S. Han Random Variables 37 representation point p. The pmf of Y P[Y = q] = f X (t)dt = 1/8 for all q. I q

39 Y. S. Han Random Variables 38 Example: Let the random variable Y be defined by Y = ax + b, where a is a nonzero constant. Suppose that X has cdf F X (x), find F Y (y). Sol: {Y y} and A = {ax + b y} are equivalent event. If a > 0 then A = {X (y b)/a}, and thus F Y (y) = P [ X y b ] a If a < 0, then A = {X (y b)/a} and F Y (y) = P [ X y b ] a = F X ( y b a ) a > 0. ( ) y b = 1 F X. a

40 Y. S. Han Random Variables 39 Therefore, we have f Y (y) = 1 a f X 1 a f X ( y b a ) ( y b a ) a > 0 a < 0 = 1 ( ) y b a f X. a

41 Y. S. Han Random Variables 40

42 Y. S. Han Random Variables 41 Example: Let X be a Gaussian random variable with mean m and standard deviation σ: f X (x) = 1 2πσ e (x m)2 /2σ 2 < x < Let Y = ax + b. Find the pdf of Y. Sol: From previous example, we have f Y (y) = 1 2π a σ e (y b am)2 /2(aσ) 2. Y also has a Gaussian distribution with mean am + b and standard deviation a σ.

43 Y. S. Han Random Variables 42 Example: Let random variable Y be defined by Y = X 2, where X is a continuous random variable. Find the cdf and pdf of Y. Sol: The event {Y y} occurs when {X 2 y} or equivalently { y X y} for y nonnegative. The event is null when y is negative. Then F Y (y) = 0 y < 0 F X ( y) F X ( y) y 0 and f Y (y) = f X( y) 2 y = f X( y) 2 y f X( y) 2 y + f X( y) 2 y. y > 0

44 Y. S. Han Random Variables 43

45 Y. S. Han Random Variables 44 Consider Y = g(x) as shown below Consider the event C y = {y < Y < y + dy}. Let B x be its equivalence in the x-axis. As shown in the figure, g(x) = y has three solutions and B x = {x 1 < X < x 1 + dx 1 } {x 2 < X < x 2 + dx 2 }

46 Y. S. Han Random Variables 45 Thus, {x 3 < X < x 3 + dx 3 }. P[C y ] = f Y (y) dy = P[B x ] = f X (x 1 ) dx 1 +f X (x 2 ) dx 2 +f X (x 3 ) dx 3. In general, we have f Y (y) = k f X (x) dy/dx = x=xk k f X (x) dx dy. x=xk

47 Y. S. Han Random Variables 46 Example : Let Y = X 2. For Y 0, the equation y = x 2 has two solutions, x 0 = y and x 1 = y. Since dy/dx = 2x, we have f Y (y) = f X( y) 2 y + f X( y) 2. y Example: Let Y = cos(x), where X is uniformly distributed in the interval (0, 2π]. Find the pdf of Y.

48 Y. S. Han Random Variables 47 Sol: Two solutions in the interval, x 0 = cos 1 (y) and x 1 = 2π x 0. dy dx = sin(x 0 ) = sin(cos 1 (y)) = 1 y 2. x0

49 Y. S. Han Random Variables 48 Since f X (x) = 1/(2π), The cdf of Y is f Y (y) = F Y (y) = = 1 2π 1 y π 1 y 2 1 π for 1 < y < 1. 1 y 2 0 y < sin 1 y π 1 y 1 1 y > 1.

50 Y. S. Han Random Variables f Y (y) y

51 Y. S. Han Random Variables Expected Value of Random Variables

52 Y. S. Han Random Variables 51 The Expected Value of X The expected value or mean of a random variable X is defined by E[X] = tf X (t)dt If X is a discrete random variable, then E[X] = k x k p X (x k ) Note that E[X] may not converge.

53 Y. S. Han Random Variables 52 The mean for a uniform random variable between a and b is given by E[X] = b a t b a dt = a + b 2 E[X] is the midpoint of the interval [a,b]. If the pdf of X is symmetric about a point m, then E[X] = m. That is, when f X (m x) = f X (m + x), we have 0 = + (m t)f X (t)dt = m + tf X (t)dt.

54 Y. S. Han Random Variables 53 The pdf of a Gaussian random variable is symmetric at x = m. Therefore, E[X] = m.

55 Y. S. Han Random Variables 54 Exercise: Show that if X is a nonnegative random variable, then E[X] = and E[X] = 0 (1 F X (t))dt if X continuous and nonnegative P[X > k] if X nonnegative, integer-valued. k=0

56 Y. S. Han Random Variables 55 Expected value of Y = g(x) Let Y = g(x), where X is a random variable with pdf f X (x). Y is also a random variable. Mean of Y is E[Y ] = + g(x)f X (x)dx.

57 Y. S. Han Random Variables 56 Variance of X Variance of the random variable X is defined by VAR[X] = E[(X E[X]) 2 ]. Standard deviation of X STD[X] = VAR[X] 1/2 measure of the spread of a distribution. Simplification VAR[X] = E[X 2 2E[X]X + E[X] 2 ] = E[X 2 ] 2E[X]E[X] + E[X] 2 = E[X 2 ] E[X] 2

58 Y. S. Han Random Variables 57 Example: Find the variance of the random variable X that is uniformly distributed in the interval [a,b]. E[X] = (a + b)/2, and VAR[X] = 1 b a b Let y = (x (a + b)/2). Then a ( x a + b ) 2 dx 2 VAR[X] = 1 b a (b a)/2 (b a)/2 y 2 dy = (b a)2 12.

59 Y. S. Han Random Variables 58 Example: Find the variance of a Gaussian random variable. Multiply the integral of the pdf of X by 2πσ to obtain + e (x m)2 /2σ 2 dx = 2πσ. Differentiate both sides with respect to σ to get + ( ) (x m) 2 e (x m)2 /2σ 2 dx = 2π. Then σ 3 VAR[X] = 1 + (x m) 2 e (x m)2 /2σ 2 dx = σ 2. 2πσ

60 Y. S. Han Random Variables 59 Properties Let c be a constant. Then VAR[c] = 0, VAR[X + c] = VAR[X], VAR[cX] = c 2 VAR[X]. nth moment of the random variable X is given by E[X n ] = + x n f X (x)dx.

61 Y. S. Han Random Variables Markov and Chebyshev Inequalities Markov Inequality Suppose X is a nonnegative random variable with mean E[X]. Then Since E[X] = P[X a] E[X] a a 0 a tf X (t)dt + a for X nonnegative tf X (t)dt af X (t)dt = ap[x a]. a tf X (t)dt

62 Y. S. Han Random Variables 61 Chebyshev Inequality Consider random variable X with E[X] = m and VAR[X] = σ 2. Then P[ X m a] σ2 a 2. Proof: Let D 2 = (X m) 2. Markov inequality for D 2 gives P[D 2 a 2 ] E[(X m)2 ] a 2 = σ2 a 2. {D 2 a 2 } and { X m a} are equivalent events.

63 Y. S. Han Random Variables Transfer Methods The Characteristic Function The characteristic function of a random variable X is defined by Φ X (ω) = E [ e jωx] = f X (x)e jωx dx, where j = 1 is the imaginary unit number. Φ X (ω) can be viewed as the expected value of a function of X, e jωx.

64 Y. S. Han Random Variables 63 Φ X (ω) is the Fourier transform of the pdf f X (x) with a reversal in the sign of the exponent. From the Fourier transform inversion formula we have f X (x) = 1 2π Φ X (ω)e jωx dω.

65 Y. S. Han Random Variables 64 Example: The characteristic function for an exponentially distributed random variable with parameter λ is given by Φ X (ω) = = 0 λ λ jω. λe λx e jωx dx = 0 λe (λ jω)x dx

66 Y. S. Han Random Variables 65 If X is a discrete random variable, we have Φ X (ω) = k p X (x k )e jωx k. If X is an integer-valued random variable, we have Φ X (ω) = k= p X (k)e jωk. The above is the Fourier transform of the sequence p X (k). It is a periodic function of ω with period 2π. By the inversion formula we have p X (k) = 1 2π 2π 0 Φ X (ω)e jωk dω k = 0, ±1, ±2,....

67 Y. S. Han Random Variables 66 Example: The characteristic function for a geometric random variable is given by Φ X (ω) = = pq k e jωk = p k=0 p 1 qe jω. ( ) qe jω k k=0

68 Y. S. Han Random Variables 67 The moment theorem states that the moments of X are given by E [X n ] = 1 d n j n dω nφ X(ω). ω=0 Proof: First we expend e jωx in power series in the definition of Φ X (ω): } Φ X (ω) = f X (x) {1 + jωx + (jωx)2 + dx. 2! Assuming that all the moments of X are finite and that the series can be integrated term by term, we have Φ X (ω) = 1 + jωe[x] + (jω)2 E [ X 2] + + (jω)n E [X n ] n! 2! +.

69 Y. S. Han Random Variables 68 If we differentiate n times and evaluate at ω = 0, we have d n dω nφ X(ω) = j n E [X n ]. ω=0

70 Y. S. Han Random Variables 69 Example: To find the mean of an exponentially distributed random variable, we differentiate Φ X (ω) = λ(λ jω) 1 once, and obtain Φ X(ω) = λj (λ jω) 2. Then E[X] = Φ X(0)/j = 1/λ.

71 Y. S. Han Random Variables 70 The Probability Generating Function The probability generating function G N (z) of a nonnegative integer-valued random variable N is defined by G N (z) = E [ z N] = p N (k)z k. k=0 G N (z) can be viewed as the expected value of a function of N, z N. G N (z) is the z-transform of the pmf p N (k) with a sign change in the exponent. Similar to the derivation of the moment theorem, we have p N (k) = 1 k! d k dz k G N(z). z=0

72 Y. S. Han Random Variables 71 d dz G N(z) = z=1 p N (k)kz k 1 z=1 = k=0 kp N (k) = E[N]. k=0 d 2 dz 2G N(z) = p N (k)k(k 1)z k 2 z=1 z=1 = k=0 k(k 1)p N (k) = E[N(N 1)] k=0 = E [ N 2] E[N]. Thus, the mean and variance of N are given by E[N] = G N(1)

73 Y. S. Han Random Variables 72 and V AR[N] = G N(1) + G N(1) (G N(1)) 2.

74 Y. S. Han Random Variables 73 Example: The probability generating function for the Poisson random variable with parameter α is given by G N (z) = k=0 α k k! e α z k = e α = e α e αz = e α(z 1). k=0 (αz) k k! The first two derivatives of G N (z) are given by G N(z) = αe α(z 1) and G N(z) = α 2 e α(z 1). Therefore, E[N] = α and V AR[N] = α 2 + α α 2 = α.

75 Y. S. Han Random Variables 74 The Laplace Transform of the pdf The Laplace transform of the pdf is given by X (s) = 0 f X (x)e sx dx = E [ e sx]. X (s) can be viewed as an expected value of a function of X, e sx. The moment theorem also holds for X (s): E [X n ] = ( 1) n dn ds n X (s). s=0