1 Variance of a Random Variable

Transcription

1 Indian Institute of Technology Bombay Department of Electrical Engineering Handout 14 EE 325 Probability and Random Processes Lecture Notes 9 August 28, Variance of a Random Variable The expectation E[X] is also nown as the first moment or mean of the random variable. Can we put some quantitative measure on how close the random variable is to its mean. In electrical engineering such situations frequently occur in sampling and quantization, where one wishes to now the quantization error around a level of finite bit representation. To put it more formally, we are trying to find the mean square error, i.e, the difference X a is squared and then expectation is taen to obtain E (X a) 2, where a is some value of interest, for example the quantized value after sampling. The variance of a random variable V ar(x) is defined as V ar(x) E (X E[X]) 2 In the notes, we will often denote V ar(x) by σx 2. A simple expansion of the RHS above results in the following. Proposition 1 E (X E[X]) 2 EX 2 (E[X]) 2. The mean EX is the point of minimum mean squared error, as the following proposition states. Proposition 2 E(X µ) 2 E(X a) 2, a R Proof: Let us maximize the RHS with respect to a. d da E(X a)2 2E(X a) 2(EX a). Thus the only optimum is at a EX, and since the double derivative is 2 everywhere, we have the desired result. Let us now introduce the Poisson distribution and compute its variance as an example. 1.1 Poisson Distribution: P oisson(λ) The Poisson distribution is over N, and has a single parameter, namely λ R +. distribution is λ λ P (X ) e! Exercise 1 Verify that the given distribution satisfies the axioms of probability. The

2 Let us compute the mean and variance of P oisson(λ). Let us now compute EX 2 EX. E(X) e λ λ 0! e λ λ 1! λ 1 e λ λ 1 ( 1)! e λ λe λ λ. The variance can now be computed as E(X) e λ ( 2 ) λ 0! e λ ( 1) λ 2! e λ λ 2 λ 2 h 2 ( 2)! λ 2. σ 2 X EX 2 EX + EX (EX) 2 λ 2 + λ λ 2 λ. 2 Expectation in terms of Probability To explain their intimate connection, we now express the expectation of a N valued random variable in terms of just the probabilities. Theorem 1 For a non-negative integer valued random variable X, Proof: E[X] P (X n). n 1 E[X] jp (X j) j N 1 {n j} P (X j) j N n1 1 {n j} P (X j) n1 j N P (X j) n1 j n n1 P (X n). To show the convenience of this property, let us consider the example of a popular random variable in the next subsection. 2

3 2.1 Geometric Random Variable- Geometric(p) This random variable taes values in the countable state space N, and the only parameter governing its realization is p [0, 1]. In relation to tossing a coin, a geometric random variable X captures the first occurrence of HEADs. The probability of HEAD occurring on the th toss and not before it is P (X ) (1 p) 1 p, 1. Example 1 Compute E[X] and E[X 2 ] if X Geometric(p). Solution: Let us find P (X ) first. P (X ) j (1 p) j 1 p p(1 p) 1 (1 p) j j0 (1 p) 1. E[X] P (X ) 1 (1 p) p. 3 Marov s Inequality To further underline that expectation contains significant information about the distribution, we state the Marov s inequality. Theorem 2 For a non-negative valued random variable X, P (X a) E[X] a. (1) Proof: While the proof can be done by expanding the summation of expectation, we resort to a more elegant method using indicator functions. Taing expectation of both sides, we get X X 1 {X<a} + X 1 {X a} X 1 {X a} a 1 {X a}. ap (X a) E[X]. 3

4 4 More On Independence Theorem 3 Consider the random vector (X 1, X 2 ) E 1 E 2. Then X 1.X 2 is a random variable. Proof: Let Y X 1.X 2, we will show that Y is a random variable. Clearly Y is discrete. Furthermore, {Y y} {X 1.X 2 y} {(ω 1, ω 2 ) X 1 (ω 1 ) x, X 2 (ω 2 ) y x x } Since X 1 and X 2 are measurable maps, the above union is well defined and an element of F. Hence Y is a random variable. Theorem 4 Consider the random variable (X 1, X 2 ) E 1 E 2, let g 1 E 1 R and g 2 E 2 R be two functions. Then g 1 (X 1 )g 2 (X 2 ) is a random variable. Proof: Since we now g 1 (X 1 ) and g 2 (X 2 ) are random variables, applying the previous theorem shows that g 1 (X 1 ).g 2 (X 2 ) is indeed a random variable. Let us now come bac to our discussion of independence. The independence of random variables is preserved by individual functional transformations. Theorem 5 Let X 1 and X 2 be independent random variables in E 1 and E 2 respectively. Consider two functions g 1 E 1 R and g 2 E 2 R. The random variables g 1 (X 1 ) and g 2 (X 2 ) are independent. Solution: The RVs g 1 (X 1 ) and g 2 (X 2 ) are discrete. Considering joint events, {g 1 (X 1 ) y 1, g 2 (X 2 ) y 2 } (x 1,x 2 ) g 1 (x 1 )y 1,g 2 (x 2 )y 2 {X 1 x 1, X 2 x 2 }, where the events inside the union are disjoint. The probability of the event on the LHS is, P (g 1 (X 1 ) y 1, g 2 (X 2 ) y 2 ) P (X 1 x 1, X 2 x 2 ) (x 1,x 2 ) g 1 (x 1 ) y 1, g 2 (x 2 ) y 2 (x 1,x 2 ) g 1 (x 1 ) y 1, g 2 (x 2 ) y 2 P (X 1 x 1 )P (X 2 x 2 ) P (X 1 x 1 ) P (X 2 x 2 ) x 1 g 1 (x 1 )y 1 x 2 g 2 (x 2 )y 2 P (g 1 (X 1 ) y 1 )P (g 2 (X 2 ) y 2 ). Thus, by the definition of independence of two random variables, g 1 (X 1 ) and g 2 (X 2 ) are independent. 4

5 Note: There is some room for an argument that the correct proof should show P (g 1 (X 1 ) y 1, g 2 (X 2 ) y 2 ) P 1 (g 1 (X 1 ) y 1 )P 2 (g 2 (X 2 ) y 2 ), where P 1 ( ) and P 2 ( ) are the corresponding measures. Indeed we have shown this desired result, but the notation that we used in the theorem is an universally accepted one. In particular, we may use the same notation P ( ) for different probability measures, but depending on the argument, the associated measure will be understood. For example, P (X 1 x 1 ) is the measure induced by X 1 on E 1, where as P (X 2 x 2 ) is a possibly different measure, induced by the random variable X 2. Observe that the transformations were applied in a separable fashion on each variable. Let us now state a simple, yet powerful law for the distribution of expectation over product of independent random variables. Proposition 3 The independent random variables X and Y whenever these expectations are well defined. Proof: E[g 1 (X)g 2 (Y )] E[g 1 (X)][Eg 2 (Y )] Eg 1 [X]g 2 [Y ] g 1 (x)g 2 (y)p (X x, Y y) (2) x,y g 1 (x)g 2 (y)p (X x)p (Y y) (3) x,y x E 1 g 1 (x)p (X x) y E 2 g 2 (y)p (Y y) (4) E[g 1 (X)]E[g 2 (Y )]. (5) Example 2 A fair die is thrown times. Let X i be the outcome of the i th toss. Consider the probability association, P (X 1 x 1,, X x ) 1 6. Show that the random variables X 1,, X are pairwise independent, i.e. every pair of X i, X j with i j are independent. Solution: Clearly each X i taes values in {1,, 6}. Let us consider the event {X i x i, X j x j }. P (X i x i, X j x j ) x 1,,x m {x i,x j } P (x 1,, x ) 1 x 1,,x m {x i,x j } Thus, P (X i x i, X j x j ) P (X i x i )P (X j x j ). See footnote 1 below. 1 we used x 1,, x {x i, x j } to tae the sum of over all subscripts except i and j 5

6 Example 3 Consider the above example, and let S X i. Compute the mean and variance of S. Solution: Let the mean be denoted as µ. µ E[S ] Let us compute the variance σ 2 S E[(S µ ) 2 ]. E[X i ] 7 2. E[(S µ ) 2 2 ] E[( (X i E[X i ])) ] (6) E[ (X i E[X i ]) 2 + (X i E[X i ])(X j E[X j ])] (7) i,j i j E(X i E[X i ]) 2 + E[(X i E[X i ])(X j E[X j ])] (8) i,j i j σx 2 i + E[X i E[X i ]]E[X j E[X j ]] (9) i,j i j σ 2 X i (10) Since σ 2 X i 70 24, i, we have σ2 S In fact, our computations in the last example show that we can generalize this result to arbitrary random variables which are pair-wise independent. Theorem 6 Let X 1,, X be a collection of pairwise independent random variables. Then σ 2 X 1 + +X σ 2 X i. We have already defined independent sequence of random variables. particular class is the most appealing. Among this, a a 4.1 IID Random Variables Definition 1 A sequence X n, n 1 of RVs is said to be Independent and Identically Distributed (IID) if 1. X n, n 1 is a independent sequence, each X n taing values in the same set E. 2. P (X i x) P (X j x), i, j and x E. 6