2 (Statistics) Random variables

Transcription

1 2 (Statistics) Random variables References: DeGroot and Schervish, chapters 3, 4 and 5; Stirzaker, chapters 4, 5 and 6 We will now study the main tools use for modeling experiments with unknown outcomes in statistics: random variables and their distribution. 2.1 Single random variables and distributions Consider an experiment in a sample space S. A random variable is a function that transforms the sample space into R, assigning a real value to each possible outcome of the experiment. In general we denote it by X(s) (s S) or simply X when there is no scope for confusion. Attached to each random variable is a probability rule that measures the likelihood of a particular outcome. Suppose A is a set in R and we wish to measure the probability that X A. This is given by: p(x A) = p(s S : X(s) A) A probability rule is generally described by a function, the cumulative distribution function, abbreviated as cdf. For any real value x, the cdf F X (x) is defined as follows: F (x) = p(x x) For a function F X defined in all R to be a cdf it must satisfy the following conditions: be non-decreasing; F X (+ ) = 1 F X () = 0 p(a < X b) = F X (b) F X (a) A random variable X is discrete or follows a discrete distribution if F can take only a finite number of different values {k 1, k 2,..., k n }. A random variable X is discrete or follows a discrete distribution if F can take only a discrete number of different values (possibly infinitely many) {k 1, k 2,..., k n,...}. A random variable X is continuous or follows a continuous distribution if it can take every value in a real interval. 1

2 2.1.1 Discrete distributions A discrete random variable takes values only on a discrete set of values, {k 1, k 2,...}. For these variables we can define a function f X called the probability density function and abbreviated as pdf : f X (x) = p(x = x) The pdf measures the likelihood of each particular outcome x. Notice: If x is not one of the possible outcomes of the experiment, {k 1, k 2,...}, then f X (x) = 0. At points in the possible space of outcomes, the pdf will be strictly positive. These are called mass points of the distribution. Thus f X is always non-negative. Since it measures the probability of each possible event, it cannot be bigger than 1. The pdf of X defines the cdf of X as follows: F X (x) = p(x x) = f X (k i ) i:k i x Since F X is a cdf, it must be that: F X is non-decreasing: if x 2 > x 1 then all values k i that are no larger then x 1 must not be larger than x 2 and thus F X (x 2 ) must be at least equal to F X (x 1 ); F X (+ ) = all i f X(k i ) = 1; F X () = 0; p(a < X b) = i:a<k i b f X(k i ) = i:k i b f X(k i ) i:k i a f X(k i ) = F X (b) F X (a). As a consequence of its definition, the cdf F X is a step function. Examples of discrete distributions: discrete uniform, bernoulli, binomial Continuous distributions A continuous random variable X can assume any value within an interval which may be bounded or not. 2

3 In this case, the cdf of X, F X may be continuous over the whole R or, at least, will be continuous over intervals of R - it may have some discontinuity points. To start with, suppose that the cdf of X is continuous and differentiable over R. Then we can define the function f X as follows: f X (x) = df X(x) dx = F X(x) (1) The function f X defined above is called the probability density function and abbreviated pdf. It measures the marginal change (increase) in F X as x changes infinitesimally. A consequence of this definition is that f X (x) is always non-negative. The reverse of (1) is that the cdf can be defined as being the function F that satisfies the following condition: F X (x) = x f X (x)dx meaning that it measures the area below the curve of f X. This definition of cdf can be extended to random variables that follow a continuous distribution in all R except possibly for a finite number of points. In this case f X (x) = F X (x) for all x where the derivative exists. Notice that from our definition the following properties follow: f X(x)dx = 1; p(x > x) = x f X (t)dt = 1 F X (x); p(a < X b) = F X (b) F X (a) = b a f X(x)dx; p(a < X b) = p(a X < b) = p(a < X < b) = p(a X b); the two properties above then imply that at a point a where the distribution of X is continuous: p(x = a) = p(a X a) = = 0 a a f X (x)dx meaning that the likelihood of the realisation of a particular value in the (continuous) distribution of X is zero. But then the function f X is non-unique: it can be changed in a discrete (finite or infinite) number of points and still form the same cdf. We solve the ambiguity by using always the continuous version of f X unless there are reasons to do it otherwise. Examples: uniform distribution, normal distribution. 3

4 2.1.3 Functions of a random variable Consider a discrete random variable first: Suppose the random variable X is defined on the space {k 1, k 2,...}, following a discrete distribution with pdf f X so that the probability of x is p(x = x) = f X (x). Now consider a transformation of X through a function h to form a new random variable Y : Y = h(x). Y follows a new probability rule f Y which is defined as follows: Now consider a continuous random variable: f Y (y) = p(y = y) = p(h(x) = y) = f X (x) x:h(x)=y X is a continuous random variable with a pdf f X (x). Consider again a transformation of X through a function h. The resulting random variable is Y = h(x). The cdf of Y can now be defined as: F Y (y) = P (Y y) = p(h(x) y) = f X (x)dx x:h(x) y Now suppose that h is strictly monotonic (either increasing or decreasing). Thus h is invertible and we can write X = h 1 (Y ). In this case, the cdf of Y is: f Y (y) = f X (h 1 (y)) dh 1 (y) dy Moments The distribution of a random variable contains all the information about it. However, it is often cumbersome and difficult to present. Instead, some functions of the random variable summarise the distribution and are often presented. The most commonly used functions are the moments of the random variable. Expected value: central moment of the distribution. 4

5 For a discrete random variable with possible realisations {k 1, k 2,..., k n } E X (X) = n k i f X (k i ) i=1 For a continuous random variable: E X (X) = xf X (x)dx What is the expected valued of Z = h(x) where X is a continuous random variable? The expected value may or may not lie at the centre of the distribution of X. Some properties of the expected value: E(c) = c where c is a constant; E(a + bx) = a + be(x) where a and b re scalars; If g(x) = g 1 (X) + g 2 (X) is a function then E(g(X)) = E(g 1 (X)) + E(g 2 (X)). If g is a non-linear function then E(g(X)) is generally different from g(e(x)). Variance: measures the dispersion of the distribution. The variance of a distribution is given by: V (X) = E [ (X E(X)) 2] = E(X 2 ) E(X) 2 We also define standard deviation to be: sd(x) = V (X) Some properties of the variance: V (c) = 0 where c is a constant; V (ax) = a 2 V (X) where a is a scalar; V (ax + b) = a 2 V (X) where a and b are scalars. Higher order moments: These help characterise a distribution. They may be centred or not: non centred moment of order k: E(X k ); centred moment of order k: E [ (X E(X)) k]. Median: another central moment of the distribution. It is the point m that divides the distribution in two parts, each with a probability of

6 The median of the distribution of a continuous random variable X is defined as follows: median(x) = m if p(x m) = p(x > m) = 0.5 The median of the distribution of a discrete random variable X is defined as the smallest value m such that: p(x m) 0.5 Quantile: the median is an example of a quantile, the 0.5-quantile. In general, the p-quantile of a distribution is the value x that divides the distribution in two parts, one with probability p and the other with probability 1 p. For a continuous random variable X, the p-quantile is defined as: Q p (X) = x if p(x x) = p For a discrete random variable X, the p-quantile is defined as the smallest x such that p(x x) p 2.2 Bivariate distributions We may imagine cases where more than one random variable is required to describe an experiment. We will now study how to deal with more than one random variable in simultaneous. Let (X, Y ) be a pair of random variables. We now want to characterise their joint distribution. Discrete case: if both X and Y are discrete random variables defined on the space S, the joint pdf is f XY (x, y) = p(x = x, Y = y) Again f XY is always non-negative and satisfies: f XY (x, y) = 1 (x,y) S The cdf is now: F XY (x, y) = x i x f XY (x i, y j ) y j y 6

7 Continuous case: if X and Y are continuous random variables, the joint cdf is F XY (x, y) = P (X x, Y y) This is a nondecreasing function in both arguments such that: F XY (, ) = 0 F XY (+, + ) = 1 We can now define the pdf to be: and thus: f XY (x, y) = 2 F XY (x, y) x y p(a x < X b x, a y < Y b y ) = bx by a x a y f XY (x, y)dydx and p(x a, Y b) = bx by = F XY (a, b) f XY (x, y)dydx Marginal distribution Consider again the case of two random variables (X, Y ). If the joint cdf is known, then the cdf of each variable can be derived. In the discrete case, this amounts to sum over all the possible values of the other variable. Let S be the support of (X, Y ) (the set of possible values that (X, Y ) may assume) and suppose there are M X and M Y different possible values that X and Y can assume, respectively. The marginal distribution of X is defined by its marginal pdf, f X, as follows: f X (x) = p(x = x) = and similarly to Y : M Y f XY (x, y j ) j=1 f Y (y) = p(y = y) = M X f XY (x i, y) i=1 7

8 In the continuous case we need to integrate over one of the variables to obtain the cdf of the other: F X (x) = F Y (y) = x y f XY (x, y)dydx f XY (x, y)dydx The marginal cdf s can now be obtained from the first derivatives of the marginal pdf: f X (x) = f Y (y) = f XY (x, y)dy f XY (x, y)dx Conditional distribution We have encountered the concept of conditional probability before. We can now apply it to distribution functions. Suppose we have a pair of random variables (X, Y ) and wish to determine the probability of some realisation of y given that we have some information about X. In particular, we can derive: p(y y, X x) P (Y y X x) = p(x x) = F XY (x, y) F X (x) This is true for both discrete and continuous random variables. For discrete random variables we can immediately write the pdf in a similar way: p(y = y, X = x) p(y = y X = x) = p(x = x) = f XY (x, y) f X (x) = f Y X (y x) and the cdf: p(y y X = x) = p(y y, X = x) p(x = x) = F Y X (y x) 8

9 For continuous random variables we need to take a small interval in X and write a similar relationship: p(y y, X x + ) p(y y, X x) P (Y y x < X x + ) = p(x x + ) p(x x) = F XY (x +, y) F XY (x, y) F X (x + ) F X (x) = [F XY (x +, y) F XY (x, y)]/ [F X (x + ) F X (x)]/ Taking the limits as approaches 0 yields p(y y X = x) = F XY (x, y)/ x df X (x)/ dx = F XY (x, y)/ x f X (x) = F Y X (y x) and we can now take the derivatives with respect to Y to obtain: In both the continuous and discrete cases: f Y X (y x) = F Y X(y x) y = f XY (x, y) f X (x) f XY (x, y) = f Y X (y x)f X (x) = f X Y (x y)f Y (y) and thus: f X Y (x y) = f Y X(y x)f X (x) f Y (y) Moments Expected value: Let g(x, Y ) be a function of the two random variables (X, Y ). Then: E XY (g(x, Y )) = { MX i=1 g(x, y)f XY (x, y)dydx for continuous random variables My j=1 g(x i, y j )f XY (x i, y j )dydx for discrete random variables But then: E XY (g 1 (X, Y ) + g 2 (X, Y )) = E XY (g 1 (X, Y )) + E XY (g 2 (X, Y )) 9

10 It is also true that: E XY (g(x)) = E X (g(x)) Covariance: cov(x, Y ) = E(XY ) E(X)E(Y ) Correlation: corr(x, Y ) = cov(xy ) / V (X)V (Y ) Independence Two random variables (X, Y ) are independent if F XY (x, y) = F X (x)f Y (y) but this implies that f XY (x, y) = f X (x)f Y (y) and f X Y (x y) = f X (x) which means that knowing one does not say anything about the other. In this case we have some results for the moments. If X and Y are independent then: E(XY ) = E(X)E(Y ) V (ax + by ) = a 2 V (X) + b 2 V (Y ) E(X Y ) = E(X) and E(Y X) = E(Y ) where E(Y X = x) = yf Y X (y x)dy Iterated expectations This is a very useful result. It states that: E X (X) = E Y [ EX Y (X Y ) ] Based on this result we can prove that, for example: if E(Y X) = 0 then E(XY ) = 0; if E(Y X) = 0 then E(Y ) = 0. 10

11 2.3 Many random variables The above results extend simply to the case where there are many random variables. In such case they are generally arranged in vectors. Let X = [X 1 X 2... X n ] be an n 1 vector of random variables. The joint distribution function is: F X (x) = p(x x) = p(x 1 x 1, X 2 x 2,..., X n x n ) The joint pdf is: f X (x) = n F X (x) x 1 x 2... x n Some of the most important moments are the following. Expected value: E X (X) = E X1 (X 1 ) E X2 (X 2 ). E Xn (X n ) Each of the expectations inside the vector are performed using the marginal distributions, so for example: E X1 (X 1 ) = =... x 1 f X1 (x 1 )dx 1 x 1 f X (x 1, x 2,..., x n )dx 1 dx 2... dx n The variance is given by: V X (X) = E(XX ) E(X)E(X) An important example is: V X (a X + b) = V X (X a) = a V (X)a This is a quadratic form. Since the variance is always non-negative, it yields that V X (a X + b) 0. But then, V (X) is psd. 11

12 2.4 Exercises 1. An exam consists of 100 multiple-choice questions. Form each question there are four possible answers, only one of them being correct. If a candidate guesses answers at random, what is the probability of getting at least 30 questions correct? 2. Two fair dices are thrown. Let X be the number of points in the first die and Y be the number of points in the second die. Define Z = X + Y and W = XY. Find the expectations and variances of X, Y, Z, W. E(W 2 ). Also find E(X 2 ), E(Y 2 ), E(Z 2 ) and 3. The pdf of a random variable X is: f(x) = { αx(2 x) if 0 < x < 2 0 otherwise Find α, E(X) and V (X). 4. Let (X, Y, Z) be independent random variables such that: E(X) = 1 and V (X) = 2 E(y) = 0 and V (Y ) = 3 E(Z) = 1 and V (Z) = 4 Let T = 2X + Y 3Z + 4 U = (X + Z)(Y + Z) Find E(T ), V (T ), E(T 2 ) and E(U). 12