2 Functions of random variables

Transcription

1 2 Functions of random variables A basic statistical model for sample data is a collection of random variables X 1,..., X n. The data are summarised in terms of certain sample statistics, calculated as functions of the random variables, one important instance being the sample mean X = n 1 n i=1 X i. In this chapter we discuss the density transformation techniques useful for calculating distributions of sample statistics. 2.1 Transformation of densities Example Let random variable X be the temperature in Celsius degrees observed in an experiment, with some known pdf f X. In Fahrenheit degrees the temperature will be Y = 1.8X What is the pdf of Y? Let us argue first in terms of the cdf s. We have F Y (y) = P(Y y) = P(1.8X + 32 y) = P(X (y 32)/1.8) = F X ((y 32)/1.8). Differentiating this equality yields f Y (y) = f X ( y 32 Proposition 2.1. Suppose rv X has pdf f X and Y = ax + b where a 0 and b R. Then the pdf of Y is 1.8 f Y (y) = f ax+b (y) = f X ( y b a ). ) 1 a. (1) This is the linear change of variables formula for densities. Note it is a and not a that appears in the formula: we do not exclude negative a, but the density must be a nonnegative function! Denoting I the support of f X, the support J of f Y is the range of the function x ax+b with domain I, that is J = {y R : (y b)/a I}. 1

2 The proof of (1) for a > 0 is exactly the same, as in the Celsius-Fahrenheit example. For a < 0 we get F Y (y) = P(Y y) = P(aX + b y) = P(X (y b)/a) = 1 P(X < (y b)/a) = 1 F X ((y b)/a). ( ) ( ) Differentiation leads to f Y (y) = f y b 1 X a a = f y b 1 X a a. Example For X Exp(λ) and a > 0 we can show that ax Exp(λ/a). Indeed, we have f X (x) = λe λx (for x 0), thus by (1) f ax (y) = f X (y/a)/a = (λ/a)e (λ/a)y. The supports are I = J = (0, ). Example If Z N (0, 1) (standard normal rv) and a 0 then and for the rv az + b we have f az+b (y) = f Z ( y b a f Z (x) = 1 2π exp ( x 2 /2 ) ) 1 a = 1 2π a exp ( ) (y b)2 2a 2 which is the normal N (b, a 2 ) density with mean b and variance a 2. We see that a general normal random variable can be obtained as a linear function of the standard normal r.v. Z N (0, 1) az + b N (b, a 2 ). The supports in this case are I = J = R. More generally, for a 0 Z N (µ, σ 2 ) az + b N (aµ + b, a 2 σ 2 ). In particular, (Z µ)/σ N (0, 1) (this transformation is sometimes called standardisation). Often we need to consider rv s obtained as nonlinear functions of a given rv, for instance e X, log X or X 2. 2

3 Example Let X Exp(λ) and Y = X 2. Both variables have support (0, ). We get for y > 0 F Y (y) = P(Y y) = P(X 2 y) = P(X y) = F X ( y). Differentiating f Y (y) = f X ( y) 1 2 y. Recalling the formula for f X we get the pdf for Y and for y < 0 we have f Y (y) = 0. f Y (y) = λe λ y 1 2 y, y > 0, To discuss a general density transformation formula, assume that X is a random variable with given pdf f X supported by interval I. Here, we understand interval in the extended sense: the interval I can be finite, halfline (e.g. (0, )) or the whole real line R. The interval may or may not include the endpoints. Let g be a function on I, whose derivative either satisfies g (x) > 0 for all x I or satisfies g (x) < 0 for all x I. In the first case g is increasing and in the second a decreasing function on I. Define another random variable Y = g(x), which is continuous with some pdf f Y and support J := {g(x) : x I}. We denote g 1 (y) the function inverse to g(x) 1. Theorem 2.1. Let g : I J be a monotone function with g (x) 0 on interval I. Then f Y relates to f X by the density transformation formula Proof. Suppose g is decreasing, then f Y (y) = f X (g 1 (y)) (g 1 ) (y). (2) F Y (y) = P(Y y) = P(g(X) y) = P(X g 1 (y)) = 1 F X (g 1 (y)). Differentianting yields f Y (y) = f X (g 1 (y))(g 1 ) (y) = f X (g 1 (y)) (g 1 ) (y). 1 Not to be confused with 1/g(y). 3

4 If g is increasing we have F Y (y) = P(Y y) = P(g(X) y) = P(X g 1 (y)) = F X (g 1 (y)), and differentiating yields (2) again. Let us connect the formula (2) with the change of variable of integration from your Calculus II course. The expectation of Y = g(x) is given by E(Y ) = g(x)f X (x)dx, I where the pdf of X is used. Using the change of variable in the integral y = g(x), x = g 1 (y), dx = (g 1 ) (y)dy this becomes E(Y ) = yf X (g 1 (y)) (g 1 ) (y) dy. Comparing with (2) we arrive at E(Y ) = E(g(X)) = J J yf Y (y)dy, which is the formula for expectation E(Y ) in terms of the pdf f Y. The transformation formula is easy to remember in simpler notation. Let us just write y(x), x(y) (in place of g(x), g 1 (y)) for the functional relation between variables x and y. Then f Y (y) = f X (x(y)) dx dy. (3) We can re-write the formula using the fact that dx dy dy = 1/ dx. The transformation formula becomes / dy f Y (y) = f X (x(y)) dx. (4) The equation y = g(x) must be solved for x in terms of y, and this value of x substituted into f X (x) and dy/dx, thus leaving the formula for f Y entirely in terms of y. 4

5 Example Let X Uniform[0, 1], that is f X (x) = 1 for x [0, 1]. Consider Y = X 2. The supports of X and Y are I = J = [0, 1]. Solving the relation y = x 2 for x Substituting in (3) yields x = y, dx dy = 1 2 y. f Y (y) = 1 2 y 1/2, y [0, 1]; this pdf is denoted Beta(1/2, 1) and is a special case of the so-called beta density. We could also use (4), this would involve computing dy/dx = 2x, then replacing x by y. The density transformation formula (2) works both ways. If Y = g(x) then X = g 1 (Y ), thus using (2) yields f X (x) = f Y (g(x)) g (x). (5) For two density functions f X and f Y equivalent: the transformation formulas are f Y (y) = f X (g 1 (y)) (g 1 ) (y) f X (x) = f Y (g(x)) g (x). (6) Example Let U Uniform[0, 1] and let V = log U (we denote log x the natural logarithm base e, same as ln x). Then I = [0, 1] and J = [0, ), because log x is a decreasing function with log x as x 0 and log 1 = 0. With substitution v = log u, u = e v, du dv = e v we see that f V (v) = f U (u(v)) du dv = 1 e v = e v, v > 0, so V Exp(1). The function u = e v is inverse to v = log u. Therefore using equivalence (6) we conclude (without calculation) that for V Exp(1) the random variable e V is uniformly distributed on [0, 1]. 5

6 We summarise this as the following uniform-exponential transformation rules U Uniform[0, 1] log U Exp(1) V Exp(1) e V Uniform[0, 1]. (7) Many-to-one functions We turn to a more complex case, where function y = g(x) is not necesssarily monotone on I, so can be many-to-one function g : I J. Example Consider Y = X 2. This corresponds to the square function y = x 2, which is nonnegative and has zero derivative at x = 0. For the cdf of Y we have for y > 0 F Y (y) = P(Y y) = P(X 2 y) = P( X y) = P( y X y) = F X ( y) F X ( y). Differentiating this identity yields f Y (y) = 1 2 y f X( y) y f X( y). Two terms here appear because the quadratic parabola has two branches, that is for each y > 0 there are two values x = ± y contributing to f Y. To formulate the density transformation result within reasonable degree of generality, we will assume that g : I J has the property that the set {x : g(x) = y} is finite for every y J. The latter holds, in particular, when g is continuously differentiable on I and there are finitely many values x I such that g (x) = 0. Let Y = g(x). Under our assumptions the density transformation formula becomes f Y (y) = / dy f X (x) dx. (8) {x:g(x)=y} The equation y = g(x) must be solved for x in terms of y, and every solution x substituted into f X (x) and dy/dx, thus leaving the formula for f Y entirely in terms of y. Let us revise the example with Y = X 2. We have x = ± y and dy/dx = 2x, therefore dy/dx = 2 y, for x = y 6

7 dy/dx = 2 y, for x = y so dy/dx = 2 y in both cases. The density transformation formula becomes f Y (y) = f X (x) 1 2x = 1 2 y f X( y) y f X( y), {x:x 2 =y} which is the same result as obtained above by using the method of cumulative distribution functions. Next is an important application to the density of a squared standard normal rv. Example Let X N (0, 1), Y = X 2. So the function is y = x 2, the supports are I = (, ), J = [0, ) and the starting density is f X (x) = ( 2π) 1 e x2 /2. We compute the transformed density as f Y (y) = f X ( y) 1 2 y + f X( y) 1 2 y = 1 e y/2 1 2π 2 y + 1 e y/2 1 2π 2 y = 1 y e 1 2 y, 2π where we recognise the Gamma(1/2, 1/2) density, also denoted χ 2 1 and called chi-square density with one degree of freedom). We summarise this finding as Z N (0, 1) Z 2 χ 2 1. The density transformation formula extends literally to functions which have no derivative at some points. An example of the latter kind appears in the next exercise. Exercise For X N (0, 1) find the pdf of X. 2.2 The probability integral transform We review the exponential-uniform example, to observe a special feature. 7

8 Example We have seen in (7) that for V Exp(1) the random variable Y = e V is uniformly distributed on [0, 1], that is Y Uniform[0, 1]. We leave as an exercise checking that for U = 1 Y the distribution is again uniform, that is U Uniform[0, 1]. This gives us the relation V Exp(1) U = 1 e V Uniform[0, 1]. But F V (v) = 1 e v is the cdf of V, therefore U = 1 e V can be written as U = F V (V ). We see that substituting exponential rv in its own cdf yields a uniformly distributed rv. This is a special case of a very general fact. Definition Let F X be the cdf of rv X. The random variable Y = F X (X) is called the probability integral transform of X. The name derives from the representation of cdf as the probability integral F X (x) = x f X(z)dz. The idea is that F X (X) always has the uniform distribution when X is a continuous rv. To avoid small complications, we assume that the support of X is some interval I (bounded or infinite), thus f X (x) > 0 for x I. Note that the range of F X is J = [0, 1], because F X (x) is some probability, so taking values between 0 and 1. The function F X has a positive derivative on I, hence strictly increasing. Therefore there exists an inverse function F 1 X. Theorem 2.2. Let X be a continuous rv with cdf F X. Then F X (X) Uniform[0, 1]. Proof. The cdf of U Uniform[0, 1] is the linear function, F U (u) = u for u [0, 1]. Since F 1 (F (u)) = u we have P(F X (X) u) = P(X FX 1 (u)) = F X(FX 1 (u)) = u. The result has a converse: if U Uniform[0, 1] then F 1 (U) F is a random variable with cdf F. This gives a practical way to simulate a random variable with given distribution. A standard random number 8

9 generator outputs a sample value u from the uniform distribution. Calculating F 1 (u) we obtain a sample value from the cdf F. Definition The inverse cdf F 1 is called the quantile function associated with cdf F. For every p (0, 1) the function outputs the value F 1 (p) = z p called the p-quantile (or 100p%-percentile of the distribution). Hence the quantile satisfies the equation F (z p ) = p. For instance, F 1 (0.5) = z 0.5 is the median of the distribution. The quantile just introduced is the lower quantile, because the probability below z p is p. The lower p-quantile is also the upper (1 p)-quantile (also called (1 p)100% upper percentile), because the probability above z p is 1 p. 2.3 Multivariate density transformation We are looking for the generalisation of the density transformation theorem for joint density of two or more random variables. Let X 1, X 2 be rv s with joint pdf f X1,X 2 (x 1, x 2 ) having support I = {(x 1, x 2 ) R 2 : f X1,X 2 (x 1, x 2 ) > 0}. Suppose the correspondence given by two functions y 1 = g 1 (x 1, x 2 ), y 2 = g(x 1, x 2 ) (9) is such that these functions have continuous partial derivatives and (y 1, y 2 ) (x 1, x 2 ) 0, where (y 1, y 2 ) (x 1, x 2 ) = det ( y1 x 1 y 2 x 2 y 2 y 1 x 1 x 2 ). 9

10 is the Jacobian (also called the Jacobian determinant). Then (9) is a bijection onto some domain J R 2 and there exists an inverse correspondence 2 x 1 = g 1 (y 1, y 2 ), x 2 = g 1 (y 1, y 2 ). (10) The Jacobian of the inverse correspondence is ( ) (x 1, x 2 ) x1 x 1 (y 1, y 2 ) = det y 1 y 2 x 2 y 1 x 2 y 2 / (y1, y 2 ) = 1 (x 1, x 2 ) Theorem 2.3. The random variables Y 1 = g 1 (X 1, X 2 ), Y 2 = g 2 (X 1, X 2 ) have the joint pdf f Y1,Y 2 (y 1, y 2 ) = f X1,X 2 (g1 1 (y 1, y 2 ), g2 1 (y 1, y 2 )) (x 1, x 2 ) (y 1, y 2 ). (11) In the formula denotes the absolute value. / It is sometimes more convenient to compute first 1 (y 1,y 2 ) (x 1,x 2 ), as a function of the variables x 1, x 2, but then (10) should be used to obtain the density formula entirely in terms of variables y 1, y 2. Example In the important special case of linear transformation we have and y 1 = a 11 x 1 + a 12 x 2 + b 1, y 2 = a 21 x 1 + a 22 x 2 + b 2, y i x j = a ij (x 1, x 2 ) (y 1, y 2 ) = 1 ( ). a11 a det 12 a 21 a 22 2 The notation does not mean that g 1 1 is inverse function to g 1 : the latter as a one-dimensional function of two variables cannot be inverted. 10

11 The method of density transformation gives the joint pdf for Y 1 = g 1 (X 1, X 2 ), Y 2 = g 2 (X 1, X 2 ). From the joint density we can find the marginal densities f Y1 and f Y2 by integration. Next example illustrates the method. Example Suppose X 1 and X 2 have joint pdf f X1,X 2 (x 1, x 2 ) = exp( (x 1 + x 2 )), x 1 0, x 2 0, which means that X 1 are X 2 are independent with the same Exp(1) density. We are interested in the distribution of the sum X 1 + X 2. We set Y 1 = X 1, Y 2 = X 1 +X 2. Consider the transformation y 1 = x 1 and y 2 = x 1 + x 2 with inverse x 1 = y 1, x 2 = y 2 y 1. This maps the positive quadrant I = (0, ) (0, ) to J = {(y 1, y 2 ) R 2 : 0 < y 1 < y 2 }. The Jacobian is ( (x 1, x 2 ) 1 0 (y 1, y 2 ) = det 1 1 So the joint pdf of Y 1 and Y 2 is given by f Y1,Y 2 (y 1, y 2 ) = e y 1 (y 2 y 1 ) = e y 2, ) = 1. 0 y 1 y 2 < The pdf Y 2 = X 1 + X 2 is calculated from f Y1,Y 2 (y 1, y 2 ) as the marginal pdf of Y 2 by integrating out the variable y 1. f Y2 (y 2 ) = f Y1,Y 2 (y 1, y 2 )dy 1 = which is the Gamma(2, 1) density. We conclude that y2 0 e y 2 dy 1 = y 2 e y 2, 0 < y 2 <, X 1, X 2 iid Exp(1) X 1 + X 2 Gamma(2, 1) In general, if X and Y are independent continuous rv s, the pdf of their sum is f X+Y (z) = f X (z y)f Y (y)dy = f X (x)f Y (z x)dx. 11

12 The density f X+Y is called convolution of the densities f X and f Y. Example We consider X 1 and X 2 with the joint pdf f X1,X 2 (x 1, x 2 ) = 8x 1 x 2, 0 < x 1 < x 2 < 1. Suppose we want to find the pdf of Y 1 = X 1 /X 2. It is convenient to introduce a complimentary variable Y 2 = X 2, since we can then easily find the inverse correspondence. Thus we deal with functions y 1 = g 1 (x 1, x 2 ) = x 1 /x 2, y 2 = g 2 (x 1, x 2 ) = x 2. The inverse correspondence between (x 1, x 2 ) and (y 1, y 2 ) becomes x 1 = y 1 y 2, x 2 = y 2. We have I = {(x 1, x 2 ) : 0 < x 1 < x 2 < 1} which implies that J = {(y 1, y 2 ) : 0 < y 1 < 1, 0 < y 2 < 1}. To see the latter, note that y 2 like x 2 varies in the limits from 0 to 1, and when y 2 is fixed the ratio y 1 = x 1 /x 2 is again between 0 and 1. The Jacobian is (x 1, x 2 ) (y 1, y 2 ) = det ( y2 y ) = y 2. So f Y1,Y 2 (y 1, y 2 ) = 8(y 1 y 2 )y 2 y 2 = 8y 1 y2, 3 (y 1, y 2 ) J. Thus the marginal pdf of Y 1 is f Y1 (y 1 ) = 1 0 8y 1 y 3 2 dy 2 = 8y 1 [ y This is the pdf for distribution denoted Beta(2, 1). ] 1 0 = 2y 1, 0 < y 1 < 1. If the correspondence which maps x 1, x 2 to y 1 = g 1 (x 1, x 2 ), y 2 = g 2 (x 1, x 2 ) is many-to-one, the joint density of Y 1, Y 2 is f Y1,Y 2 (y 1, y 2 ) = {(x 1,x 2 ):g 1 (x 1,x 2 )=y 1,g 2 (x 1,x 2 )=y 2 } f X1,X 2 (x 1, x 2 ) (y 1, y 2 ) (x 1, x 2 ) 1. It will be easy now to extend the density transformation method to n jointly distributed continuous random variables. We say that random variables X 1,..., X n are jointly distributed, if their values are observed in the 12

13 same random experiment. The joint distribution is described by the joint pdf f X1,...,X n (x 1,..., x n ), which is a function depending on (x 1,..., x n ) R n that allows us to calculate probabilities by integration in n dimensions P((X 1,..., X n ) A) = f X1,...,X n (x 1,..., x n )dx 1 dx n, A A R n. Next is the most important special case: The iid sequence Random variables X 1,..., X n are iid (independent, identically distributed) with (one-dimensional) pdf f if their joint density is the product form f X1,...,X n (x 1,..., x n ) = f(x 1 )f(x 2 ) f(x n ). (12) Note that f = f Xi is the marginal density for every X i, i = 1,..., n. In Statistics, it is common to call the observed values of X 1,..., X n a sample from the population described by the pdf f. Vector notation and density transformation in higher dimensions To generalise the transformation formula to higher dimensions it is very convenient to use the vector notation. Definition A n-dimensional random variable is a random vector X = X 1. X n whose components X 1,..., X n are one-dimensional random variables. Using the vector notation we write f X (x) for the joint pdf of X 1,..., X n. Let I := {x R n : f X (x) > 0} be the support of f X and suppose that g : I J is a bijection from I to some other domain J R n. Suppose g has continuous partial derivatives. Consider the n-dimensional random variable Y = g(x), which means that we consider n 13

14 one-dimensional random variables Y 1 = g 1 (X 1,..., X n ), Y 2 = g 2 (X 1,..., X n ),. Y n = g n (X 1,..., X n ). Introduce the Jacobians y 1 y x (y) 1 1 x n (x) = det..... = y n y x 1 n x n x 1 x y 1 1 y n det x n y 1 1 / / (x) (y) = 1 x n y n The vector notation allows us to formulate the density transformation theorem in the form most close to the case of one dimension. Theorem 2.4. Let Y = g(x) where g is as above and suppose that (y) (x) 0 for x I. Then the pdf of Y is f Y (y) = f X (g 1 (y)) (x) (y) = f X(g 1 (y)) (y) (x) for y J (and f Y (y) = 0 for y / J). If the formula with (y) (x) 1 is used, the variable x in the computed Jacobian should be expressed in terms of y by solving the system of equations y = g(x) for x. Example Let Y = AX + b be a random vector obtained as multidimensional linear transformation of X, where A is a n n square matrix with det A 0 and b R n is a fixed vector. In this notation, AX stays for the product of a matrix and a column vector, thus in coordinate-wise writing n Y i = a ij X j + b i, i = 1,..., n. j=1 The Jacobian of the vector-function y = xa + b is (y) (x) = det A 14 1

15 because y i / x j = a ij. Solving y = Ax + b for x yields x = A 1 (y b), where A 1 is the matrix inverse to A thus f Y (y) = f X (A 1 (y b)) det A 1. Compare with the analogous formula in the one-dimensional case. Many-to-one vector functions More generally, let g : I J be a many-to-one function, such that the set {x : g(x) = y} is finite for each y. Then for Y = g(x) the density transformation theorem generalises as f Y (y) = {x:g(x)=y} f X (x) (y) (x) 1, y J. To apply this formula, one should find all solutions x to the vector equation g(x) = y which, written in coordinates, is a system of n equations in unknowns x 1,..., x n. For each solution the Jacobian must be calculated and the result left entirely in terms of the variables y 1,..., y n. 15