Elements of Probability Theory

Transcription

1 Short Guides to Microeconometrics Fall 2016 Kurt Schmidheiny Unversität Basel Elements of Probability Theory Contents 1 Random Variables and Distributions Univariate Random Variables and Distributions Bivariate Random Variables and Distributions Moments Expected Value or Mean Variance and Standard Deviation Higher order Moments Covariance and Correlation Conditional Expectation and Variance Random Vectors and Random Matrices 9 4 Important Distributions Univariate Normal Distribution Bivariate Normal Distribution Multivariate Normal Distribution Version: , 09:06

2 Elements of Probability Theory 2 1 Random Variables and Distributions A random variable is a variable whose values are determined by a probability distribution. This is a casual way of defining random variables which is sufficient for our level of analysis. For more advanced probability theory, a random variable will be defined as a real-valued function over some probability space. In section 1 to 3, a random variable is denoted by capital letters, e.g. X, whereas its realizations are denoted by small letters, e.g. x. 1.1 Univariate Random Variables and Distributions A univariate discrete random variable is a variable that takes a countable number K of real numbers with certain probabilities. The probability that the random variable X takes the value x k among the K possible realizations is given by the probability distribution P (X = x k ) = P (x k ) = p k with k = 1, 2,..., K. K may be in some cases. This can also be written as p 1 if X = x 1 p 2 if X = x 2 P (x k ) =. p K if X = x K Note that K p k = 1. k=1 A univariate continuous random variable is a variable that takes a continuum of values in the real line. The distribution of a continuous random variable X can be characterized by a density function or probability

3 3 Short Guides to Microeconometrics density function (pdf) f(x). The nonnegative function f(x) is such that P (x 1 X x 2 ) = x2 x 1 f(x)dx. defines the probability that X takes a value in the interval [x 1, x 2 ]. Note that there is no chance that X takes exactly the value x, P (X = x) = 0. The probability that X takes any value on the real line is f(x)dx = 1. The distribution of a univariate random variable X is alternatively described by the cumulative distribution function (cdf) F (x) = P (X < x). The cdf of a discrete random variable X is F (x) = P (X = x k ) = p k. x k x x k x and of a continuous random variable X F (x) = F (x) has the following properties: x f(t)dt F (x) is monotonically nondecreasing F () = 0 and F ( ) = 1. F (x) is continuous to the left 1.2 Bivariate Random Variables and Distributions A bivariate continuous random variable is a variable that takes a continuum of values in the plane. The distribution of a bivariate continuous random variable (X, Y ) can be characterized by a joint density function

4 Elements of Probability Theory 4 or joint probability density function, f(x, y). The nonnegative function f(x, y) is such that P (x 1 X x 2, y 1 Y y 2 ) = x2 y2 x 1 y 1 f(x, y)dydx defines the probability that X and Y take values in the interval [x 1, x 2 ] and [y 1, y 2 ], respectively. Note that f(x, y)dydx = 1. The marginal density function or marginal probability density function is given by such that f(x) = f(x, y)dy P (x 1 X x 2 ) = P (x 1 X x 2, Y ) = x2 x 1 f(x)dx. The conditional density function or conditional probability density function with respect to the event {Y = y} is given by f(y x) = provided that f(x) > 0. Note that f(x, y) f(x) f(y x)dy = 1. Two random variables X and Y are called independent, if and only if If X and Y are independent, then: f(y x) = f(y) f(x, y) = f(x) f(y) P (x 1 X x 2, y 1 Y y 2 ) = P (x 1 X x 2 ) P (y 1 Y y 2 )

5 5 Short Guides to Microeconometrics More generally, if a finite set of n continuous random variables X 1, X 2, X 3,..., X n are mutually independent, then f(x 1, x 2, x 3,..., x n ) = f(x 1 ) f(x 2 ) f(x 3 )... f(x n ). 2 Moments 2.1 Expected Value or Mean The expected value or mean of a discrete random variable with probability distribution P (x k ) and k = 1, 2,..., K is defined as K E[X] = x k P (x k ) k=1 if the series converges absolutely. The expected value or mean of a continuous univariate random variable with density function f(x) is defined as E[X] = if the integral exists. xf(x)dx For a random variable Z which is a continuous function φ of a discrete random variable X, we have: K E[Z] = E[φ(X)] = φ(x k )P (x k ) k=1 For a random variable Z which is a continuous function φ of the continuous random variables X and Y, we have: E[Z] = E[φ(X)] = E[Z] = E[φ(X, Y )] = φ(x)f(x)dx φ(x, y)f(x, y)dx dy

6 Elements of Probability Theory 6 The following rules hold in general, i.e. for discrete, continuous and mixed types of random variables: E[α] = α E[αX + βy ] = αe[x] + βe[y ] E [ n i=1 X i] = n i=1 E[X i] E[X Y ] = E[X] E[Y ] if X and Y are independent where α R and β R are constants. 2.2 Variance and Standard Deviation The variance of a univariate random variable X is defined as V[X] = E [ (X E[X]) 2] = E[X 2 ] (E[X]) 2 The variance has the following properties: V[X] 0 V[X] = 0 if and only if X = E[X] The following rules hold in general, i.e. for discrete, continuous and mixed types of random variables: V[αX + β] = α 2 V[X] V[X + Y ] = V[X] + V[Y ] + 2Cov[X, Y ] V[X Y ] = V[X] + V[Y ] 2Cov[X, Y ] V [ n i=1 X i] = n i=1 V[X i] if X i and X j independent for all i j where α R and β R are constants. Instead of the variance, one often considers the standard deviation σ X = VX.

7 7 Short Guides to Microeconometrics 2.3 Higher order Moments The j-th moment around zero is defined as E [ (X E[X]) j]. 2.4 Covariance and Correlation The Covariance between two random variables X and Y is defined as: Cov[X, Y ] = E [ (X E[X])(Y E[Y ]) ] = E[XY ] E[X]E[Y ] = E [ (X E[X])Y ] = E [ X(Y E[Y ]) ] The following rules hold in general, i.e. for discrete, continuous and mixed types of random variables: Cov[αX + γ, βy + µ] = αβcov[x, Y ] Cov[X 1 + X 2, Y 1 + Y 2 ] = Cov[X 1, Y 1 ] + Cov[X 1, Y 2 ] + Cov[X 2, Y 1 ] + Cov[X 2, Y 2 ] Cov[X, Y ] = 0 if X and Y are independent where α R, β R, γ R and µ R are constants. The correlation coefficient between two random variables X and Y is defined as: ρ X,Y = Cov[X, Y ] σ X σ Y where σ X and σ Y denote the corresponding standard deviations. The correlation coefficient has the following property: 1 ρ X,Y 1 The following rule holds: ρ αx+γ,βy +µ = ρ X,Y ρ X,Y = 0 if X and Y are independent where α R and β R are constants.

8 Elements of Probability Theory 8 We say that X and Y are uncorrelated if ρ = 0 X and Y are positively correlated if ρ > 0 X and Y are negatively correlated if ρ < Conditional Expectation and Variance Let (X, Y ) be a bivariate discrete random variable and P (y k X) the conditional probability of Y = y k given X. Then the conditional expected value or conditional mean of Y given X is K E[Y X] = E Y X [Y ] = y k P (y k X). Let (X, Y ) be a bivariate continuous random variable and f(y x) the conditional density of Y given X. Then the conditional expected value or conditional mean of Y given X is E[Y X] = E Y X [Y ] = k=1 yf(y X)dy. The law of iterated means or law of iterated expecations holds in general, i.e. for discrete, continuous or mixed random variables: [ ] E X E[Y X] = E[Y ]. The conditional variance of Y given X is given by V[Y X] = E [ (Y E[Y X]) 2 ] [ ] X = E Y 2 X (E[Y X]) 2. The law of total variance is [ ] [ ] V[Y ] = E X V [Y X] + VX E[Y X].

9 9 Short Guides to Microeconometrics 3 Random Vectors and Random Matrices In this section we denote matrices (random or non-random) by bold capital letters, e.g. X and vectors by small letters, e.g. x. Let x = (x 1,..., x n ) be a (n 1)-dimensional vector such that each element x i is a random variable. Let X be a (n k)-dimensional matrix such that each element x ij is a random variable. Let a = (a 1,..., a n ) be a n 1-dimensional vector of constants and A a (m n) matrix of constants. The expectation of a random vector, E[x), and of a random matrix, E[X), summarize the expected values of its elements, respectively: E[x 1 ] E[x 11 ] E[x 12 ]... E[x 1k ] E[x 2 ] E[x 21 ] E[x 22 ]... E[x 2k ] E[x] = and E[X] = E[x n ] E[x n1 ] E[x n2 ]... E[x nk ] The following rules hold: E[a x] = a E[x] E[Ax] = AE[x] E[AX] = AE[X] E[tr(X)] = tr(e[x]) for X a quadratic matrix The variance-covariance matrix of a random vector, V(x), summarizes all variances and Covariances of its elements: V[x] = E [ (x E[x]) (x E[x]) ] = E[xx ] (E[x])([E[x]) V[x 1 ] Cov[x 1, x 2 ]... Cov[x 1, x n ] Cov[x 2, x 1 ] V[x 2 ]... Cov[x 2, x n ] = Cov[x n, x 1 ] Cov[x n, x 2 ]... V[x n ]

10 Elements of Probability Theory 10 The following rules hold: V[a x] = a V[x] a V[Ax] = A V[x] A where the (m n) dimensional matrix A with m n has full row rank. If the variance-covariance matrix V[x] is positive definite (p.d.) then all random elements and all linear combinations of its random elements have strictly positive variance: V [a x] = a V [x] a > 0 for all a 0. 4 Important Distributions 4.1 Univariate Normal Distribution The density of the univariate normal distribution is given by: f(x) = 1 σ 2π e 1 2( x µ σ ) 2. The normal distribution is characterized by the two parameters µ and σ. The mean of the normal distribution is E[X] = µ and the variance V[X] = σ 2. We write X N(µ, σ 2 ). The univariate normal distribution with mean µ = 0 and variance σ 2 = 1 is called the standard normal distribution N(0, 1). 4.2 Bivariate Normal Distribution The density of the bivariate normal distribution is 1 f(x, y) = 2πσ X σ Y 1 ρ 2 { [ (x ) 2 ( ) 2 ( ) ( ) ]} 1 µx y µy x µx y µy exp 2(1 ρ 2 + 2ρ. ) σ X σ Y σ X σ Y

11 11 Short Guides to Microeconometrics If (X, Y ) follows a bivariate normal distribution, then: The marginal densities f(x) and f(y) are univariate normal. The conditional densities f(x y) and f(y x) are univariate normal. E[X] = µ X, V[X] = σ 2 X, E[Y ] = µ Y, V[Y ] = σ 2 Y. The correlation coefficient between X and Y is ρ X,Y = ρ. E[Y X] = µ Y + ρ σ Y σ X (X µ X ) and V[Y X] = σ 2 Y (1 ρ2 ). The above properties characterize the normal distribution. It is the only distribution with all these properties. Further important properties: If (X, Y ) follows a bivariate normal distribution, then ax + by is also normally distributed: ax + by N(µ X + µ Y, σ 2 X + σ 2 Y + 2ρσ X σ Y ). The reverse implication is not true. If X and Y are bivariate normally distributed with Cov[X, Y ] = 0, then X and Y are independent. 4.3 Multivariate Normal Distribution Let x = (x 1,..., x n ) be a n-dimensional vector such that each element x i is a random variable. In addition let E[x] = µ = (µ 1,..., µ n ) and V[x] = Σ with σ 11 σ σ 1n σ 21 σ σ 2n Σ = σ n1 σ n2... σ nn where σ ij = Cov[x i, x j ].

12 Elements of Probability Theory 12 A n-dimensional random variable x is multivariate normally distributed with mean µ and variance-covariance matrix Σ, x N(µ, Σ) if its density is: ( f(x) = (2π) n/2 (det Σ) 1/2 exp 1 ) 2 (x µ) Σ 1 (x µ). Let x N(µ, Σ) and A a (m n) matrix with m n and m linearly independent rows then we have Ax N(Aµ, AΣA ). References Amemiya, Takeshi (1994), Introduction to Statistics and Econometrics, Cambridge: Harvard University Press. Hayashi, Fumio (2000), Econometrics, Princeton: Princeton University Press. Appendix A. Stock, James H. and Mark W. Watson (2012), Introduction to Econometrics, 3rd ed., Pearson Addison-Wesley. Chapter 2.