3 Operations on One Random Variable - Expectation

Transcription

1 3 Operations on One Random Variable - Expectation 3.0 INTRODUCTION operations on a random variable Most of these operations are based on a single concept expectation. Even a probability of an event can be rewritten as an expectation. How? 3.1 EXPECTATION the mathematical expectation of X the expected value of X the mean value of X the statistical average of X E[X], X >> Example Expected Value of a Random Variable three Pythagorean means of positive real numbers x 1, x 2,..., x N the arithmetic mean N i=1 x i N the geometric mean (Π N i=1x i ) 1 N 1

2 the harmonic mean N N i=1 1 x i In certain situations, especially many situations involving rates and ratios, the harmonic mean provides the truest average. For instance, if a vehicle travels a certain distance at a speed x (e.g. 60 kilometres per hour) and then the same distance again at a speed y (e.g. 40 kilometres per hour), then its average speed is the harmonic mean of x and y (48 kilometres per hour), and its total travel time is the same as if it had traveled the whole distance at that average speed. (Wikepedia) an alternate form for the arithmetic mean of {x i } i I 1 x {i I : x i = x} = x {i I : x i = x} I I x R x R where I is a finite index set. Motivated by a relative frequency interpretation of probability, we define the statistical mean or the expected value of a discrete random variable X as 기대값 = 값곱하기확률 의합, i.e., E[X] = cf. the center of gravity (1-D version): For point mass, N i=1 x im i N i=1 M i N x i p X (x i ). (3.1-1) i=1 2

3 where x i is the location and M i is the mass at x i. For distributed mass, xm(x)dx m(x)dx where x is the location and m(x) is the density. Stieltjes Integral Form: xdm(x) dm(x) where M(x) is the cumulative density function. Def. The expected value of a random variable X is defined, in terms of its cdf, as X = E[X] where the integral is a Stieltjes integral. xdf X (x), In terms of the sample space S and the random variable X(s) as a function on S, E[X] X(s)dP (s). For a continuous random variable X, For a discrete random variable X, E[X] = f X (x) = S xf X (x)dx (3.1-2) p X (x i )δ(x x i ) (3.1-3) i=1 3

4 Thus, using the pdf we again have E[X] = xf X (x)dx = x i p X (x i ) (3.1-4) i=1 >> Example If a random variable s density is symmetrical about a line x = a, E[X] = a if f X (x + a) = f X ( x + a) (3.1-5) The expected value of X must be interpreted as When the number of independent observations of X increases, the arithmetic average becomes very close to the statistical average. This is called the Strong Law of Large Numbers (SLLN) X 1 + X X N lim N N Expected Value of a Function of a Random Variable E[X] almost surely A function is the simplest model of an electrical system. What is the output of a function when the input is modeled as a random variable? What is the expected value of the output of a function when the input is modeled as a random variable? If a random variable Y is defined as g(x), where g( ) is a real function and X is a real random variable, then E[Y ] = yf Y (y)dy where f Y (y) is the pdf of Y. How can we find f Y (y) from f X (x)? 4

5 X is a random input to a system whose I/O relation described by g( ). Y is the output of the system. The pdf of the input is known. Find the expected valued of the output. (Fundamental Theorem of Expectation) E[Y ] equals to E[g(X)], which can be calculated as E[g(X)] = g(x)f X (x)dx (3.1-6) Thus, we don t need to find f Y (y) if the integration in terms of f X (x) is mathematically tractable. For a discrete random variable X, E[g(X)] = N g(x i )p X (x i ) discrete random variable (3.1-7) i=1 >> Example Expectation is a linear operator. E[aX] = ae[x] E[X 1 + X 2 ] = E[X 1 ] + E[X 2 ]: (X 1 (s) + X 2 (s))dp (s) = X 1 (s)dp (s) + X 2 (s)dp (s) >> Example entropy of a discrete source Q. Find the integral form of the relative entropy or Kullback-Leibler distance from f X (x) to f Y (y) defined as { D(f X f Y ) E log f } X(X) f Y (X) in terms of the pdf s f X (x) and f Y (y) of X and Y, respectively. 5

6 Expectation is more general than probability, F X (x) = E[1 (,x] (X)]. *Conditional Expected Value the conditional expected value of X given an event B had occurred: E[X B] xf X (x B)dx (3.1-8) If B = {X b} < b < (3.1-9) then f X (x) b f X (x X b) = f x < b X(x)dx (3.1-10) 0 x b Thus, E[X X b] = b xf X(x)dx b f X(x)dx (3.1-11) the total expectation theorem E[X] = n E[X A n ]P (A n ) where A nn is a partition. 6

7 3.2 MOMENTS two types of moments moments about the origin moments about the mean Moments about the Origin the nth order (non-central) moment of X: m 0 = 1, m 1 = X g(x) = X n n = 0, 1, 2,... (3.2-1) m n = E[X n ] = If X is a non-negative random variable, i.e., P (X 0) = 1, then E[X n ] = 0 x n f X (x)dx (3.2-2) nx n 1 (1 F X (x))dx Integrate (x n (1 F X (x))) to prove. To find E[X] = 0 (1 F X(x))dx, we just need the ccdf. Plat nx ( n 1) and the ccdf together. They provide a hint about the existence of the nth moment. 7

8 Central Moments g(x) = (X X) n n = 0, 1, 2,... (3.2-3) the nth order central moment of X: µ 0 = 1, µ 1 = 0 µ n = E[(X X) n ] = (x X) n f X (x)dx (3.2-4) Variance and Skew the mean m 1 = E[X] the variance µ 2 σx 2 = µ 2 = E[(X X) 2 ] = the standard deviation of X: the positive root of variance variance in terms of the first and second moments (x X) 2 f X (x)dx (3.2-5) σ 2 X= E[X 2 2 XX + X 2 ] = E[X 2 ] 2 XE[X] + X 2 = E[X 2 ] X 2 = m 2 m 2 1 (3.2-6) >> Example

9 the skew of the density function µ 3 = E[(X X) 3 ] the skewness of the density function = the coefficient of skewness µ 3 /σx 3 >> Example the kurtosis µ 4 /σ 4 X or µ 4/σ 4 X 3 (excess kurtosis) Chebychev s Inequality Recommended Order of Study: See Markov s inequality first. Then, study Chebychev s inequality. Then, study Chernoff s inequality. Motivated by the Chernoff bound, study the moment generating function. Finally, study the characteristic function. Let Y = X X, Then, Y is a nonnegative random variable. So, Markov s inequality leads to P (Y ɛ) = P { X X ɛ} = E[ X X ]/ɛ a twist: P { X X ɛ} = P (Y ɛ) = P (Y 2 ɛ 2 ). Then, Markov s inequality leads to P { X X ɛ} σ 2 X/ɛ 2 (3.2-7) 9

10 Alternately, P { X X ɛ} = But since X ɛ f X (x)dx + X+ɛ f X (x)dx = x X ɛ f X (x)dx (3.2-8) σ 2 X= ɛ 2 (x X) 2 f X (x)dx x X ɛ must be true, so... an alternative form of Chebychev s inequality x X ɛ (x X) 2 f X (x)dx f X (x)dx = ɛ 2 P { x X ɛ} (3.2-9) P { X X < ɛ} 1 (σ 2 X/ɛ 2 ) (3.2-10) >> Example Markov s Inequality the Markov inequality for a nonnegative random variable X, i.e., P (X 0) = 1: u(x a) x/a, x 0 = E[u(X a)] E[X/a], X 0. P {X a} E[X]/a a > 0 (3.2-11) 10

11 3.3 FUNCTIONS THAT GIVE MOMENTS two functions to calculate moments the characteristic function the moment generating function *Characteristic Function the characteristic function of a random variable X Φ X (ω) E[e jωx ] (3.3-1) Φ X (ω) is the Fourier transform (with the sign of ω reversed) of f X (x). Φ X (ω) = a function of the real variable < ω < j 1 Properties of Φ X (ω) Φ X (ω) always exists. f X (x)e jωx dx (3.3-2) Φ X (ω) fully characterizes the statistical property of X because f X (x) can be found from the inverse Fourier transform (with the sign of x reversed) of Φ X (ω). f X (x) = 1 Φ X (ω)e jωx dω (3.3-3) 2π 11

12 The nth moment of X can be found from Φ X (ω). m n = ( j) n dn Φ X (ω) dω n Φ X (ω) can be written in terms of the nth moments. [ ] Φ X (ω) = E[e jωx (jωx) n ] = E = n! >> Example n=0 (3.3-4) ω=0 (jω) n E[X n ] n! n=0 Φ X (ω) Φ X (0) = 1 (3.3-5) *Moment Generating Function the moment generating function of a random variable X a function of the real variable < ν < M X (ν) = M X (ν) = E[e νx ] (3.3-6) m n = dn M X (ν) dν n f X (x)e νx dx (3.3-7) (3.3-8) ν=0 the moment generating function may not exist for all random variables and all values of ν >> Example

13 *Chernoff s Inequality and Bound Chernoff s inequality For any ν > 0, u(x a) e ν(x a). Thus, for any ν > 0, P {X a} = E[u(X a)] E[e ν(x a) ] = e νa M X (ν) We may choose ν to minimize the right side. The minimum value is called Chernoff s bound. Q(x) e x2 2, x TRANSFORMATIONS OF A RANDOM VARIABLE input X, output Y the I/O mapping rule, transformation, system, black box, transfer characteristic T ( ) Y = T (X) (3.4-1) Figure T ( ) can be linear, nonlinear, segmented, staircase, etc. f X (x) or F X (x) is known. Determine f Y (y) or F Y (y). The key idea is to use F Y (y) = Pr(Y y) = Pr(T (X) y) = Pr(X x : T (x) y) 13

14 Deriving the result at a glance is difficult in general. We need to partition the set of transformations. Try first transforms they are easy to handle to learn lessons. Then, finally extend to the general case. Monotonic Transformations of a Continuous Random Variable A transformation T is called monotonically increasing if T (x 1 ) < T (x 2 ) for any x 1 < x 2. It is monotonically decreasing if T (x 1 ) > T (x 2 ) for any x 1 < x 2. strictly monotonically increasing, monotonically increasing, monotonically decreasing, strictly monotonically decreasing monotonically increasing, monotonically non-decreasing, monotonically non-increasing, monotonically decreasing For monotonically increasing, continuous, differentiable at all values of x for which f X (x) 0, y 0 = T (x 0 ) or x 0 = T 1 (y 0 ) (3.4-2) the cdf of Y at y 0 F Y (y 0 ) = P {Y y 0 } = P {X x 0 } = F X (x 0 ) (3.4-3) y0 f Y (y)dy = x0 =T 1 (y 0 ) f X (x)dx (3.4-4) 14

15 Figure

16 the pdf of Y at y 0 Since this result applies for any y 0 f Y (y 0 ) = f X [T 1 (y 0 )] dt 1 (y 0 ) dy 0 (3.4-5) more compactly, f Y (y) = f X [T 1 (y)] dt 1 (y) dy (3.4-6) f Y (y) = f X (x) dx dy (3.4-7) For monotonically decreasing, continuous, differentiable at all values of x for which f X (x) 0, the cdf of Y at y 0 the pdf of Y or simply F Y (y 0 ) = P {Y y 0 } = P {X x 0 } = 1 F X (x 0 ) (3.4-8) f Y (y) = f X [T 1 (y)] dt 1 (y) dy (3.4-9) f Y (y) = f X (x) dx dy (3.4-10) >> Example A linear transformation of a gaussian random variable produces another gaussian random variable. 16

17 Nonmonotonic Transformations of a Continuous Random Variable A transformation may not be monotonic. Figure F Y (y 0 ) = P {Y y 0 } = P {x Y y 0 } = f Y (y 0 ) = d dy 0 {x Y y 0 } {x Y y 0 } f X (x)dx (3.4-11) f X (x)dx (3.4-12) 17

18 f Y (y) = n f X (x n ) dt (x) dx x=xn where x n, n = 1, 2,..., are the real solutions of the equation (3.4-13) y = T (x). (3.4-14) The above explanation is hard to understand. What about case study Fig ? We assume the transformation is a piecewise strictly monotone function. Notice that F Y (y 0 ) = F X (x 3 ) F X (x 2 ) + F X (x 1 ). Now, differentiate LHS w.r.t. y 0 to obtain the pdf. You need the results from the monotone functions. >> Example Transformation of a Discrete Random Variable X is discrete. Y = T (X) is a continuous transformation. If the transformation is monotonic, there is a one-to-one correspondence between X and Y. f X (x) = n F X (x) = n P (x n )δ(x x n ) (3.4-15) P (x n )u(x x n ) (3.4-16) 18

19 Figure

20 f Y (y) = n f Y (y) = n P (y n )δ(y y n ) (3.4-17) P (y n )u(y y n ) (3.4-18) If T is not monotonic,... y n = T (x n ) (3.4-19) P (y n ) = P (x n ) (3.4-20) >> Example COMPUTER GENERATION OF ONE RANDOM VARIABLE two activities in research analysis synthesis, design two activities in transformation of random variable (analysis) Given the pdf of the input and a mapping rule, find the pdf of the output. (synthesis, design) Given the pdf s of the input and the output, find a deterministic mapping rule. existence uniqueness Why do we need synthesis? A digital computer is often used to simulate systems prior to the actual construction of the system. 20

21 Performance evaluation/estimation requires generation of random numbers. A software library may not contain the random number generator of a desired distribution. How to generate a random variable with a specified probability distribution, given another random variable s random number generator code? specialize: Start from a given uniform random number generator generalize: Learn lessons and find a solution for general cases How to generate a random variable with a specified probability distribution, given a uniform random number generator code? discrete random variable continuous random variable mixed random variable X: the output of a uniform random number generator on (0, 1) T ( ): transformation to be designed Y = T (X): the random variable of desired distribution If T (X) is monotonically nondecreasing Since X is uniform, F X (x) = x when 0 < x < 1. Thus, F Y [y = T (x)] = F X (x) (3.5-1) >> Example >> Example y = T (x) = FY 1 (x) 0 < x < 1 (3.5-2) 21

22 The inverse of the desired cdf must be available. The inverse cannot be found analytically for a gaussian random variable. Then, how? See Section 5.6. >> Example If X is uniform on [0, 1] and Y = FY 1 (X), then the cdf of Y is F Y (y). Thus, we need to find a transformation from a random variable to a uniform random variable. If the cdf of Y is F Y (y) and Z = F Y (Y ), then Z is uniform on [0, 1], because Z = F Y (Y ) = F Y (FY 1 (X)) = X. First convert X to a uniform random variable by Y = F X (X). Second, convert the uniform random variable Y to a desired random variable by Z = F Z (Y ). Draw a block diagram!!!! 3.6 SUMMARY expectation relative frequency interpretation the fundamental theorem for expectation SLLN moments cental vs. non-central moments the Markov inequality, the Chebychev inequality, the Chernoff inequality the characteristic function and moment generating function 22

23 Figure

24 Figure

25 the moment generating function the characteristic function transformation of one random variable into another finding distribution and density function of new random variable monotone function differentiable function generation of a specified random variable from a uniform r.v. to a desired r.v. from a continuous r.v. to a uniform r.v. 25