Random Variables and Expectations

Transcription

1 Inside ECOOMICS Random Variables Introduction to Econometrics Random Variables and Expectations A random variable has an outcome that is determined by an experiment and takes on a numerical value. A procedure that can be repeated and has a well defined set out outcomes is known as an experiment. If for example we were to roll a die 20 times we know that the number 3 may appear from 0 to 20 times. We may find that in the first trial of 20 rolls the number 3 appear 5 times. In the next trial of the experiment the number 3 may appear 9 times. The number of times the number 3 appears is a random variable. The random variable X can take on a value within the set {1,2,3,4,5,6, }. In general notation we denote a collection of random variables X 1, X 2,, X k with the corresponding outcomes particular outcomes of x 1, x 2,, x k. For example we could collect the yearly income from 100 randomly chosen people from the population. In this case the random variable X 1 would represent first person and the outcome or income as number would be x 1. Discrete Random Variables: X can only take on a finite number of values Continuous Random Variable: X can take on infinite values Discrete Random Variable A discrete random variable takes on only a finite number of values. The simplest example of a discrete variable is a Bernoulli random variable. The Bernoulli random variable requires that we only need to know the probability it takes on the one value. For example if you were to toss a coin and give heads the value one, then the probability that X equals one is a half P(X = 1) = 1/2. As the probabilities must sum to one it must be the case that (X = 0) = 1/2. There are many situations where the probabilities may not be half. For instance the probability a person pays a traffic infringement fine by a certain date is 0.75 and the probability they do not pay the fine before the date is More generally X can take on k possible discrete values {x 1, x 2,, x k } with the probabilities {p 1, p 2,, p k }, where p 1 + p p k = 1. Given the probability distribution function of any discrete random variable allows us to compute the probability of any event. A random variable X can take on the values 5 to +5 with equal probability therefore the probability that X = 6 will be 1/11. f(x) [0,1]; f(x) x = 1; Pr(X = x) = f(x) f X (x) = f(x) = P (X = x) f X (6) = f(6) = P (X = 6) Random Variable X pdf 1/11 1/11.. 1/11 1

2 Inside ECOOMICS From the graphs below we can see that the probability distribution is flat as all values are equally likely. Pr x Probability Disribution Conversely, the probability that X is not equal to 6 will be 1 minus 1/11. In general terms we write this below as: f(x) [0,1]; f(x) x 2 = 1; Pr(X x) = 1 f(x) Suppose that a X is the number of marks a student can score on a given exam question with three values {0,1,2,3} and assume the pdf of X is given as the following: f(0) = 0.15, f(1) = 0.40, f(2) = 0.20 and f(3) = 0.25 From these probability we can determine the probability of getting at least 1 mark as P(X 1) = P(X = 1) + P(X = 2) + P(X = 3) = = To work out this we could also have just calculated P(X 1) = 1 P(X 1) = = The pdf allows us to work out the probability of any outcome because the sum of the probabilities must equal to 1. Continuous Random Variables A continuous random variable takes on any real value with zero probability. The reason for zero probability is because as the random variable X can take on some many values the probability of any one of these outcomes is zero. In practice measurements are always discrete but random variables that take on numerous values are best treated as continuous. A good example to think of is that of prices where we refine the price to cents, however in theory we could list all the possible values imaginable even though the list may continue indefinitely. Technically the price of a good is a discrete variable, however because there are so many possible values it is unfeasible to use the properties of discrete random variables. When working with a continuous random variable it is easier to use the cumulative distribution function cdf. pdf f X (x) = f(x) = Pr (x X x + )

3 Inside ECOOMICS x cdf F X (x) = F(x) = Pr(X x) = f(y)dy For the discrete random variable, the cdf is obtained by summing the pdf over all values x i x. For a continuous random variable, F(x) is the area under the pdf, f, and to the left of the point x. This means that if x 1 < x 2 then P(X x 1 ) P(X x 2 ), that means that F(x 1 ) F(x 2 ). Therefore a cdf is either an increasing, or not decreasing, function of x. Useful Properties of cdfs for any w, P(X > w) = 1 F(c) for any numbers a < b, P(a < X b) = F(b) F(a) For a continuous random variable X, P(X c) = P(X > c) and, P(a < X < b) = P(a Xb) = P(a X < b) = P(a < X b) F X (x) = F(x) = P(X x) = f(y) Random Variable X 5 cdf 1/11 2/ /11 11/11 y x Cumlative Probability Disribution Pr PDF μ 3

4 Inside ECOOMICS Joint Probability Distributions For discrete variable X and Y the joint distribution of (X, Y) is described by the joint probability distribution function of (X, Y). We shall not cover the details of joint pdfs for continuous variables as it is outside the scope of this document. This is the probability that X = x and Y = y f (X,Y) (x, y) = P(X = x, Y = y) Random variables X and Y are independent if, and only if, f (X,Y) (x, y) = f X (x) f Y (y) for all x and y where f X is the pdf of X and f Y is the pdf of Y When X and Y are discrete random variables we can write the following, P(X = x and Y = y) = P(X = x, Y = y) = P(X = x)p(y = y) Example 1 Joint Distributions: Consider an exam with only two questions, letting X = 1 if the the student gets the first anwser correct and Y = 1 if the student anwsers the second question correctly. Assume the probability of getting the first and second question correct are 0.5 and 0.8 respectivley. If we believe that these X and Y are independent we can find the probability of getting both questions correct as. P(X = 1, Y = 1) = P(X = 1)P(Y = 1) = (0.5)(0.8) = 0.4 This means that there is a 0.4 chance of the student getting both questions correctly. This calculation will be invalid if X and Y are dependent. Conditional Probability Distribution It is often the case that we are trying to find how one random variable Y is related to one or more other random variables. The relationship of how X affects Y is given by the conditution distribution of Y given X. For discrete random variables this will become f (Y X) (y x) = f x,y(x, y), for x where f f x (x) x (x) > 0 f (Y X) (y x) = P(Y = y X = x) When Y is a continuous random variable the conditional distribution is difficult to interpret as a probability as the conditional probabilities are calculated as the area under the conditional pdf. If X and Y are independent then the value of X will not reveal any information about the value of Y. Independence means that f (Y X) (y x) = f y (y) and f (X Y) (x y) = f x (x) Conditional Distributions: We can examine the conditional probability of rain given the temperature (cold or not cold). The conditional density is Rain o Rain Cold f (Y X) (1 1) = 0.65 f (Y X) (0 1) = 0.35 ot Cold f (Y X) (1 0) = 0.3 f (Y X) (0 0) = 0.7 4

5 Inside ECOOMICS This means that the probability of rain given that it is cold is 0.65 and the probability that it does not rain when it is cold is These probabilities must sum to one. Additionally the chance of getting rain given the temperature is not cold is 0.3 and the probability that it does not rain when it is not cold is 0.7. This implies that Y and X are not independent. We are still able to find P(X = 1, Y = 1) ( the probability of it cold and raining) provided we know P(X = 1). If we assume the probability of the temperature being cold is 0.5 then we have the following, Example 1: Conditional Distributions P(X = 1, Y = 1) = P(Y = y X = x)p(x = 1) = (0.65)(0.5) = Consider tossing of a coin where X = head and Y = heads previous toss.the probability of getting heads is independent of the result of the last coin toss because X and Y are independent. Y = heads, and X = heads previous toss Head Tail heads previous f (Y X) (1 1) = 0.5 f (Y X) (0 1) = 0.5 toss tails previous toss f (Y X) (1 0) = 0.5 f (Y X) (0 0) = 0.5 f (Y X) (Y X) = 0.5 = f y (y) = 0 Therefore the outcome of a coin toss is independent of the outcome of previous coin toss. Features of Probability Distributions Mean or Expected Value (1 st Moment) The long run average of the random variables is one of the most important concepts in statistics and econometrics. The expected value of a random variable is a weighted average of the all possible values of the variable, where the weights are determined by the probability density function. Mean of Discrete Random Variable E[X] = (x) x Example 2: Expected Value Pr(X = x) = x(x) f(x) = μ X Values for X Probability E(X) = 1(0.1) + 3(0.6) + 0(0.3) = 1.9 The Mean of Continuous Random Variable E[X] = x f(x)dx = μ X If X is a continuous random variable the expectation is defined as an intergral, however it can still be interpreted as a weighted average. For most continuous distributions the expected value of X is a possible value of X. If g(x) where g is a function, then if X is a random variable it must be the case that the function g(x) is also a random variable. This means that the expectation of a function of a random variable is also a random variable. 5

6 Inside ECOOMICS Mean of a function of a Discrete Random Variable E[g(X)] = g(x) x μ X Pr(X = x) = g(x) x f(x) = Mean of a function of a Continuous Random Variable E[g(X)] = g(x) f(x)dx = μ X Example 3: Expected Value Let g(x) = x 3 Values for X g(x) = x Probability Properties of The Expected Value E(X) = 2(0.1) + 0(0.6) + 3(0.3) = 0.7 Property Expected Value 1: For any constant c, E(c) = c Property Expected Value 2: E[Y] = a + be[x] Proof E[Y] = a + be[x] E[Y] = y Pr(Y = y) = y f(y) = (a + bx)f(x) = af(x) + bxf(x) y y x = a f(x) + b xf(x) = a(1) + be[x] E[Y] = a(1) + be[x] Property Expected Value 3: The expected value of the sum is the sum of the expected values E a i X i = a i E(X i ) 6

7 Inside ECOOMICS Measures of Variability: Variance and Standard Deviation It is possible that random variables have the same means based on the pdf but do not have identical distributions. For example two random variables may have an equal mean of μ but the variance around this mean can be quite different. Pr PDF f(x) f(y) μ It is clear that although both variables have a mean of μ the distribution of X is more tightly centered about the mean when compared to Y. VAR X = E[(X i μ X ) 2 ] = p i (X i μ X ) 2 2 = σ X i 1 VAR X = E[Z 2 ] = (x μ X ) 2 f(x)dx = σ X 2 Property of the Variance 1: The Variance of a constant is zero If P(X = c) = 1 where c is a constant the variance will be Var(X) = 0 Property of the Variance 2 Var[Y] = Var(a + bx) = b 2 Var[X] Var[Y] = b 2 Var[X] Var[Y] = E Y i μ y 2 = Y i μ y 2 f(y) = a + bx a + bμ y 2 f(y) = 1 + bx (1 + bμ x ) 2 f(x) = b(x μ x ) 2 f(x) 7 = b 2 (x μ x ) 2 = Var[Y] = b 2 Var[X]

8 Inside ECOOMICS The Sums of Random Variables (Variance) Property of Variance 3: For constants a and b, Var(aXbY = a 2 Var(X) + b 2 Var(Y) + 2abCov(X, Y) If X and Y are uncorrelated then Cov(X, Y) = 0 then Var(X + Y) = Var(X) + Var(Y) Var(X Y) = Var(X) + Var(Y) Please note that the variance of the difference is the sum of the variance, not the difference in the variance. Property of Variance 4: If {X 1, X 2,, X n } are pairwise uncorrelated random variables and {a 1, a 2,, a } are constants 2 Var a i X i = a i Var(X i ) If a i = 1 for all i then for pairwise uncorreltated random variables the sum of the variances is equal to the variance of the sum. (Random variables are pairwise uncorrelated random variables if each variable in the set is uncorrelated with all the other variables, Cov X i X j = 0 i j) Standard Deviation The standard deviation is the positive square root of the variance σ(x ) = VAR X It is often preferred to work with the standard deviation rather than the variance as the standard deviation scales by the same factor as the expected value whilst the variance does not. It may be the case that the variance of X is one million times larger than the variance of Y. For example the E(X) = 5, Std(X) = 3 and Y = 100X the E(Y) = 500 and Std(Y) = 100(3) = 300. The variance in this case would be Std(Y) 2 = Var(Y) = = For this reason the standard deviation can often be more intuitive when working with variables measured in units. Property of Standard Deviation 1: The standard deviation of any constant is zero If P(X = c) = 1 where c is a constant the variance will be σ(x) = 0 Property of Standard Deviation 2: Std[Y] = Std(a + bx) = b Std[X] Covariance σ XY Covariance measures the linear dependence between two random variables. Cov(X, Y) = E( X E(X) Y E(Y) = x y(x E[X])(y E[Y])P(X = x, Y = y) = σ XY 8

9 Inside ECOOMICS σ XY = E(X μ x )(Y μ y ) The covariance between two random variables is the measure of the relationship between variables that describe a population. If E(X) = μ x and E(Y) = μ y then if X > μ x and Y > μ y (X μ x ) Y μ y > 0 if X < μ x and Y < μ y (X μ x ) Y μ y > 0 if X > μ x and Y < μ y (X μ x ) Y μ y < 0 if X < μ x and Y > μ y (X μ x ) Y μ y < 0 If σ XY > 0 then on average if X > μ x, then Y > μ y and vice versa. A positive correlation indicates that two random variables move in the same direction and whilst a negative correlation indicates that they move in opposite directions. Property of Covariance 1: If X and Y are independent then the covariance is zero (σ XY = 0) Please note that the converse is false, so that zero covariance between two variables does not imply they are independent. For example variables such as X and Y = X 2 will have Cov(X, Y) = 0, but they are clearly not indpendent because once we know X we know Y. Property of Covariance 2: Where a, b, c nd d are constants This means that the covariance between two random variables can be scaled by a constant. This is often useful in economics when we apply units of measurement to a covariance but do not want to change its meaning. Property of Covariance 3: Cauchy-Schwartz Inequality Cov(X, Y) Std(X)Std(Y) This means that the absolute value of the covariance is bounded by the product of their standard deviations. The Correlation Coefficient σ XY σ X σ Y = Corr(X, Y) = Cov(X, Y) Std(X)Std(Y) = σ XY σ X σ Y x y(x E[X])(y E[Y])P(X = x, Y = y) x (x E[X])Pr(X = x) y(y E[Y])Pr(Y = y) 9 = ρ XY If we want to measure the relationship between two variables by using the covariance we have the problem that the covariance uses the value of measurement. For instance we often use dollars (000's ect), hours worked and education in years and the covariance is sensitive to the unit of measurement used. The correlation coefficient overcomes this problem. From the equation it is

10 Inside ECOOMICS easily to see that the correlation will be equal to 0 if and only if the covariance is equal to 0. Additionally if X and Y are independent then Corr(X, Y) = 0, however this does not imply independence as the correlation coefficient, like the covariance, is a measure of linear dependence. Property of Correlation 1: 1 Corr(X, Y) 1 If Corr(X, Y) = 0 there is no linear relationship If Corr(X, Y) = 1 there is a perfect linear relationship Y = a + bx where b > 0 is some constant If Corr(X, Y) = 1 there is a perfect negative linear relationship Y = a + bx where b < 0 is some constant ote: Just because the correlation coefficient equals to zero this does not mean there is no dependence between variables. Correlation coefficient only looks at linear dependence. Variables can be related in other ways such as a v-shaped symmetric relationship. Also a relationship may only exist at extreme values for one of the variables. Property Correlation 2: The correlation coefficient does not depend on the units of measurement For constants a 1, b 1, a 2 and b 2 If a 1 a 2 > 0 If a 1 a 2 < 0 Corr(a 1 X + b 1, a 2 Y + b 2 ) = Corr(X, Y) Corr(a 1 X + b 1, a 2 Y + b 2 ) = Corr(X, Y) Conditional Expectation Conditional expectations allow us to compute the expected value of Y, given that we know the outcome of Y. The conditional expectation allows us to summarize the relationship between Y and X. The conditional expectation of Y is the weighted average of all the possible values of Y which is the same as the unconditional expectation. However now the weights are based on the fact that X is a certain value. In a sense the expectation is a function of x, where the function tells us how the expected value of Y varies with x. Essentially the conditional expectation is a random variable and a function of the conditioning variables. We write the expected of Y as E(Y X = x) or E(Y x) k E[Y X = x] = y i f (Y X) y j x = xp(x = x Y = y) j=1 When Y is continuous E(Y X) requires the intergration of y i f (Y X) (y x) over all possible values. Property of Conditional Expectation (1): E[c(X) X] = c(x), for any function c(x) Property of Conditional Expectation (2): E(a(X)Y + b(x) X) = a(x)e(y X) + b(x) An example of a conditional expectation of a function 3XY + 3X 3 E(3XY + 3X 3 X) = 3XE(Y X) + 3X 3 10

11 Inside ECOOMICS Property of Conditional Expectation (3): If X and Y are independent then E(Y X) = E(Y) Below is a proof showing that E(X Y) = E(X) This property means that if X and Y are independent then expected value of Y given X is the expected value of Y because X does not affect the value of Y. For instance the eye colour of a woman should not be related to the number of children she has. Therefore E(# children eye colour) = E(# children) Conditional Expectation Property 4: E[E(Y X)] = E(Y) This property states that if we have get E(Y X) as a function of X and take the expected value of this, then we derive the expected value of Y that is E(Y). For demonstration we may be interested in the relationship between the yearly wage and years of work experience. As an example let X = years of experience and let Y = wage. ow suppose the expected wage given years of work experience is denoted as follows, Wage = 20, (Years of experience) Y = X E(Y X) = E(wage years experience) = E 20, (years experience) 11

12 Inside ECOOMICS If E(years experience) = 10 then we can use the law of iterated expectations as follows, E 20, (years experience) = 20, E(years experience) = 20, (10) 20, (10) = 70,000 Basically there are two steps where you first find the expected value of Y given X E(Y X) then you find the expectation of X, E(X). In our example the first step was E(wage years experience) and then the second step was E(years experience). Conditional Expectation Property 5: If E(Y X) = E(Y) then Cov(X, Y) = 0 and Corr(X, Y) = 0 This means that X and Y are independent random variables and knowing the value of either one will not tell give the expected value of the other. This implies that Xand Y are uncorrelated, however the reverse of this property is not true. So it could be the case that X and Y are uncorrelated the expected value of Y could still depend on X. For example if Y = X 2 and E(Y X) = X 2 then obviously X 2 is a function of X. The expectation captures the nonlinear relationship between X and Y that correlation and covariance ignore. There is also the possibility that X and X 2 are uncorrelated and have a zero covariance. Conditional Expectation Property 6: if E(Y 2 ) < and E[g(X) 2 ] < for some function g, then E{[Y μ(x)] 2 X} E{[Y g(x)] 2 X} and E{[Y μ(x)] 2 } E{[Y g(x)] 2 } This means that the conditional mean is better than any other function of X for predicting Y. By looking at the equations we can see that we the squared error of the mean is less than the squared error of the function g. The conditional mean minimizes the unconditional expected square prediction represented in the second inequality. This is an important property for forecasting as it gives us a prediction with the smallest distance or error. Conditional Variance The variance of Y conditional on X = x is the variance associated with the conditional distribution of Y given X = x. Var(Y X = x) = E(Y 2 x) [E(Y x)] 2 For example let Y = investment and X = income then, Var(Investment Income) = (Income) This means that as the variance of income increases the variance of investment also increases. Property of Conditional Variance 1: If X and Y are independent Var(Y X) = Var(Y) Some Probability Examples Probability with more than one random variable Joint Pr Cold (X = 0) ot Cold (X = 1) SUM Distribution Snowing(Y = 0) ot Snowing(Y = 1) SUM

13 Inside ECOOMICS If it is cold what is the probability that it is snowing? Pr = = Suppose there is no snow, what is the probability it is cold? Marginal Probability Distributions Cold (X = 0) ot Cold (X = 1) Pr = = Snowing (Y=0) 0.27 ot Snowing (Y=1) 0.73 Conditional Probability Distribution Pr(X = x Y = y) Joint Pr Distribution Cold (X = 0) ot Cold (X = 1) SUM Pr(X = x Y = 0) 0.20/ / Pr(X = x Y = 1) 0.10/ / From above we can work out the following E[X Y = 0] = = 0.37 E[X Y = 1] = = E[X] = 0(0.3) + 1(0.7) = 0.7 Important Theorems 13