Topic 2: Probability & Distributions. Road Map Probability & Distributions. ECO220Y5Y: Quantitative Methods in Economics. Dr.

Transcription

1 Topic 2: Probability & Distributions ECO220Y5Y: Quantitative Methods in Economics Dr. Nick Zammit University of Toronto Department of Economics Room KN3272 n.zammit utoronto.ca November 21, 2017 Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Road Map Probability & Distributions Key Concepts: 1 Definitions (Set Theory & Rules, Marginal Probability, Joint Probability, Conditional Probability, Expectation Operator) 2 Tables/Plots (Contingency Tables, Probability Trees, Normal Probability, Area of Standard Normal, Area of Student s t) 3 Probability Distributions (Uniform, Geometric, Binomial, Poisson, Student-t, χ 2, F-dist.) 4 Ideas (Law of Large Numbers, Independence vs. Mutual Exclusivity, Bayes Rule, Chebychev s Formula & Rule, Normal Approximations) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

2 Basic Definitions Probability P(A) is the probability of an event A occuring. Sample Space S is the set of all possible outcomes Event A is an event made up of a subset of outcomes from the sample space Outcome C i is the value of a specific outcome i, i ɛ (1,.., N) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Set Theory Union A B is the set of all elements which are in A or B or both Intersection A B is the set of all elements which are in both A and B Complement A c is the set of all elements which are not in A Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

3 Set Rules 1 A B = B A 2 A B = B A 3 A (B C) = (A B) C 4 A (B C) = (A B) C 5 A = 6 A = A 7 (A c ) c = A 8 A (B C) = (A B) (A C) 9 A (B C) = (A B) (A C) 10 (A B) c = A c B c 11 (A B) c = A c B c Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Empirical vs. Theoretical Probability Theoretical (Classical) Probability The mathmatical/model based likelihood of an event If N outcomes are equally likely and N A is the number of outcomes in A, the theoretical probability of event A is P(A) = N A N Empirical (Frequentist) Probability The long-run relative frequency of an event. Guaranteed by Law of Large Numbers If n A is the number of times the event A occurred in n trials, the empirical probability of event A is P(A) n A n Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

4 Empirical Probability and Law of Large Numbers Law of Large Numbers The long-run relative frequency of repeated, independent events homes in on empirical probability as number of trials increases Proves that empirical probability is the same as theoretical probability as n This is easy to see with a simple example! Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Empirical Probability and Law of Large Numbers Percent Dice Roll Random Dice Rolls (10000 Obs) Dice Rolls (10 Obs) Random Random Dice Rolls (10 Obs) Random Dice Rolls (100 Obs) 0 10 Percent Dice Roll Percent Dice Roll Random Dice Rolls (1000 Obs) Percent Dice Roll Percent 20 Random Dice Rolls (10000 Obs) Dice Roll 5 6 Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

5 Basic Probability Rules Range Rule (Rule 1) A probability is a number between 0 and 1 For any event A, 0 P(A) 1 Probability Assignment Rule (Rule 2) The probability of the set of all possible outcomes must be 1 P(S) = 1 Complement Rule (Rule 3) The probability of an event A occurring is 1 minus the probability that it does not occur (A complement) P(A) = 1 P(A c ) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Basic Probability Concepts Marginal Probability For two events A and B, the marginal probability is the empirical probability based on total frequency of one event. Joint Probability For two events A and B, the joint probability is the empirical probability based on frequency of both events occurring together. Conditional Probability For two events A and B, the conditional probability is the empirical probability based on frequency of one event given a condition on the other event. Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

6 Advanced Probability Rules Multiplication Rule (Rule 4) For two independent events A and B, the probability that both A and B occur is the product of the individual probabilities. P(A B) = P(A) x P(B) General Multiplication Rule (Rule 7) For two events A and B, the probability that both A and B occur is the probability of one event occuring times the probability of the other occurring conditional on the first. P(A B) = P(A) x P(B A) = P(B) x P(A B) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Advanced Probability Concepts Formal Independence Events A and B are independent when the probability of either event is the same as the probability of that event conditional on the other event. P(A B) = P(A) and P(B A) = P(B) This can be extended to 3 or more events P(A B, D) = P(A) and P(B A, D) = P(B) and P(D A, B) = P(D) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

7 Advanced Probability Rules Addition Rule (Rule 5) For two disjoint events A and B, the probability that either A or B occurs is the sum of the individual probabilities. P(A B) = P(A) + P(B) General Addition Rule (Rule 6) For two events A and B, the probability that either A or B occurs is the sum of the probability each event occurs less the probability that both events occur. P(A B) = P(A) + P(B) P(A B) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Advanced Probability Concepts Mutual Exclusivity Generally the events A and B are said to be mutually exclusive events if they share no common elements P(A B) = Independent vs. Disjoint If two events are disjoint (mutually exclusive) they cannot be independent P(A B) = P(A) x P(B) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

8 Advanced Probability Concepts Bayes Rule Consider an event B and i mutually exclusive events A i : P(A i B) = P(B A i)p(a i ) j P(B A j)p(a j ) (1) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 What makes a valid probability distribution? For a density function p(x): 1 p(x) is non-negative for all values in the range of x-axis values 2 The total area under the density function p(x) in the range of x-axis values is equal to 1 S p(x)dx = 1 Note for a valid density function: If p(x) is the p.d.f. of a continuous random variable x, then the probability that x belongs to A, where A is some interval, is given by the integral of p(x) over that interval P(xɛA) = A p(x)dx If p(x) is the p.d.f. of a discrete random variable x P(xɛA) = A p(x) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

9 Probability Distributions Example: Univariate Distribution Highest Education Obtained no quals high school high school associate degree professional masters phd Total drop-out diploma diploma Notice this is a valid probability density function: p(x) is non-negative for all possible categories All the area in the density function sums to one p(x) = = 1 So if we have A = {no quals, high school drop-out} then we can calculate P(A) = = Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Probability Distributions Example: Bivariate Distribution Highest Education Obtained Gender no high school high school associate degree professional masters phd Total quals drop-out diploma diploma Female Male Total So now if we have A = {no quals, high school drop-out} and B = {male} then we can calculate P(A B) = = Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

10 Probability Trees Tree Diagrams A tree shows sequences of events as paths that look like branches of a tree Allows comparison of multiple scenarios based on multivariate probability distributions Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 The Expectation Operator Expectations for Distributions 1 E[X ] = x xp(x) = µ or E[X ] = x xp(x)dx 2 E[(X E[X ]) 2 ] = V [X ] = x (x µ)2 p(x) or E[(X E[X ]) 2 ] = x (x µ)2 p(x)dx 3 E[(X E[X ])(Y E[Y ])] = COV [X, Y ] = x y (x µ x)(y µ y )p(x, y) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

11 The Expectation Operator Laws of Expected Values 1 E[c] = c 2 E[X + c] = E[X ] + c 3 E[cX ] = ce[x ] 4 E[X + Y ] = E[X ] + E[Y ] 5 E[a + bx + cy ] = a + be[x ] + ce[y ] Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 The Expectation Operator Laws of Variance 1 V [c] = 0 2 V [X + c] = V [X ] 3 V [cx ] = c 2 V [X ] 4 V [a + bx + cy ] = b 2 V [X ] + c 2 V [Y ] + 2(b)(c)COV [X, Y ] Laws of Covariance 1 COV [X, c] = 0 2 COV [a + bx, c + dy ] = (b)(d)cov [X, Y ] Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

12 Special Probability Distributions The Bernoulli Distribution Generated by an experiment called a Bernoulli Trial leading to only two outcomes (success, failure) Bernoulli Trial has probability of success equal to p P(X = 1) = p Bernoulli Trial has probability of failure equal to 1 p P(X = 0) = 1 p = q Distribution has the following central tendencies: E[X ] = 0(q) + 1(p) = p V [X ] = E[X 2 ] E[X ] 2 = p p 2 = p(1 p) = pq Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Special Probability Distributions The Geometric Distribution Generated by having independent Bernoulli Trials until success The probability of the first success in x Bernoulli Trials has probability P(X = x) = p(q) x 1 Distribution has the following central tendencies: E[X ] = xp(x) = 1p + 2pq + 3pq = 1 p i=0 V [X ] = E[X 2 ] E[X ] 2 = q+1 p 2 1 p 2 = q p 2 Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

13 Permutations & Combinations If there are n objects to be arranged in order then how many ways can you do this? np n = n(n 1)(n 2)...1 = n! (2) If there are n objects to arrange and you must choose and order r of them how many ways can you do this? np r = n(n 1)(n 2)...(n r + 1) = n! (n r)! (3) If there are n objects to arrange and you must simply choose r of them how many ways can you do this? nc r = n P r r! = n! r!(n r)! (4) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Special Probability Distributions The Binomial Distribution Generated by having n independent Bernoulli Trials The probability of x successes in n Bernoulli Trials has probability P(X = x) = ( n C x ) p x q n x Distribution has the following central tendencies: E[X ] = n E[X i ] = p + p p = np i=0 V [X ] = n V [X i ] = p = pq + pq pq = npq i=0 Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

14 Finding Probability of Binomial Distributions Using the equation: P(X = x) = ( n C x ) p x q n x Assume n = 5 and p = 0.5 P(X = 0) = ( 5 C 0 ) = P(X = 1) = ( 5 C 1 ) = P(X = 2) = ( 5 C 2 ) = P(X = 3) = ( 5 C 3 ) = P(X = 4) = ( 5 C 4 ) = P(X = 5) = ( 5 C 5 ) = P(X 1) = P(X = 0) + P(X = 1) = = Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Finding Probability of Binomial Distributions Using a Binomial table: n = 4, X n = 5, X p Assume n = 5 and p = 0.5 P(X 1) = Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

15 Special Probability Distributions The Poisson Distribution Generated given we have time divided into n unit time intervals where independent events occur (arrive) with a mean rate of occurance The probability x independent events occur given a mean rate of occurrence λ during n is P(X = x) = e λ λ x x! Distribution has the following central tendencies: E[X ] = ( ) xp(x) = e λ 1λ 1! + 2λ2 2! + 3λ3 3! +... = e λ λe λ = λ x V [X ] = E[X 2 ] E[X ] 2 = (λ 2 + λ) λ 2 = λ Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Finding Probability with Poisson Distributions Using the equation: P(X = x) = e λ λ x x! Assume λ = 6 for our chosen n time interval P(X = 0) = e ! = P(X = 1) = e ! = P(X = 2) = e ! = P(X = 6) = e ! = P(X = 8) = e ! = P(X = 12) = e ! = P(X 1) = P(X = 0) + P(X = 1) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

16 Finding Probability of Poisson Distributions Using a Poisson table: X λ Assume λ = 6 P(X 1) = Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Special Probability Distributions The Uniform Distribution (Discrete) Generated over n equally likely outcomes The probability outcome x occurs given n possible outcomes is P(X = x) = 1 n Distribution has the following central tendencies: E[X ] = x xp(x) = 1 1 n n n 1 n = 1 n V [X ] = E[X 2 ] E[X ] 2 = 1 n n(n+1) 2 = n+1 2 n(n+1)(2n+1) 6 (n+1)2 4 = N Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

17 Special Probability Distributions The Uniform Distribution (Continuous) Generated over an equally likely interval of outcomes (a to b) The probability density function p(x) is: a x b 1 b a 0 otherwise The probability x is in the interval c and d is P(c x d) = d c b a Distribution has the following central tendencies: E[X ] = b a xp(x)dx = 1 b a b a xdx = 1 b a [ ] x 2 b 2 = b+a a 2 V [X ] = E[X 2 ] E[X ] 2 = (b2 +a 2 )+ab 3 (b+a)2 4 = (b a)2 12 Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Finding Probability with Uniform Distributions Using the equation: P(c x d) = d c b a Assume a = 0 and b = 6 P(2 x 3) = = 1 6 P(2 x 4) = = 2 6 P(0 x 6) = = 1 Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

18 Special Probability Distributions The Normal Distribution The probability density function p(x) is ( ) p(x) = (2πσ 2 ) 1/2 exp (x µ) 2 where < x < 2σ 2 The probability x is in the interval c and d is d ( ) P(c X d) = (2πσ 2 ) 1/2 exp (x µ) 2 dx 2σ 2 c Distribution has the following central tendencies: ( ) E[X ] = (2πσ 2 ) 1/2 exp (x µ) 2 dx = µ 2σ 2 V [X ] = E[X 2 ] E[X ] 2 = σ 2 Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Special Probability Distributions The Standard Normal Distribution Transform the normal distribution by standardizing: Z = X µ σ The probability density function p(z) is ( ) p(z) = (2π) 1/2 exp (x) 2 where < x < The probability x is in the interval c and d is P(c X d) = P( c µ σ X µ σ d µ σ ) 2 Distribution has the following central tendencies: E[Z] = 0 V [Z] = E[Z 2 ] E[Z] 2 = 1 Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

19 Empirical Rules Chebychev s Rule For any empirical distribution with mean x and standard deviation s, at least 100(1 1 2k )% of the data lie within k standard deviations around the mean, for k > Rule (The Empirical Rule) If a data set has an approximately bell-shaped relative frequency histogram (is approximately normally distributed): 1 Approximately 68% of the data lie within one standard deviation of the mean ( x ± s) 2 Approximately 95% of the data lie within two standard deviations of the mean ( x ± 2s) 3 Approximately 99.7% of the data lie within three standard deviations of the mean ( x ± 3s) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Finding Probability with Normal Distributions Using the equation: P(Z c) = (2π) 1/2 c exp ( (z) 2 2 ) dz P(c Z d) = P(Z d) P(Z c) = d ( ) exp (z) 2 c dz (2π) 1/2 exp (2π) 1/2 P(c X d) = P( X µ σ 2 ( (z) 2 d µ X µ σ ) P( σ c µ σ ) 2 ) dz This requires software or a very non-trivial calculation by hand! Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

20 Finding Probability with Standard Normal Distributions Using a standard normal table: Area to left of z z Assume X N(3, 16) Z N(0, 1) P(X 0) = P(Z ) = P(Z 0.75) = P(X 1) = P(Z ) = P(Z 0.5) = P(0 X 1) = P(Z 0.5) P(Z 0.75) = = Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Special Probability Distributions The Chi-Squared Distribution The probability density function is the squared standard normal χ 2 1 N(0, 1)2 The sum of independent χ 2 1 distributions can be generalised If W i χ 2 1 and W i are independent then where n are the degrees of freedom of χ 2 n χ 2 n has the following central tendencies: E[X ] = n V [X ] = E[X 2 ] E[X ] 2 = 2n n i=1 W i = χ 2 n Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

21 Finding Probability with Chi-Squared Distributions Using a Chi-Squared Density table: n = 1 (χ 2 1 ) Area to left of x x Assume n = 1 χ 2 1 P(X 2.71) =.90 P(X > 2.71) = 1 P(X 2.71) = = 0.1 Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Special Probability Distributions Student s-t Distributions t-distribution is ratio of a standard normal to a chi-squared t n N(0,1) χ 2 n /n where n are the degrees of freedom of χ 2 n As the sample size becomes large enough t becomes normal As n we get t n N(0, 1) t n has the following central tendencies: E[X ] = 0 V [X ] = E[X 2 ] E[X ] 2 = n (n 2) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

22 Finding Probability with Student s-t Distributions Using a Student-t Density table: n = 10 (t 10 ) Area to left of t t Assume n = 10 t 10 P(T 0.75) = P(T 0.5) = N(0,1) χ 2 10 /10 P( 0.75 T 0.5) = = Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Special Probability Distributions F-Distributions F-Distribution is the ratio of two independent chi-squared F m,n χ2 m /m χ 2 n /n where m & n are degrees of freedom of the numerator and denominator respectively As the sample size becomes large enough F becomes χ 2 m As n we get F m,n = χ2 m /m χ 2 / χ2 m/m F m,n has the following central tendencies: E[X ] = n (n 2) for n > 2 V [X ] = E[X 2 ] E[X ] 2 = 2n2 (m+n 2) m(n 2) 2 (n 4) Dr. Nick Zammit (UofT) Topic 2 November 21, / 47

23 Approximations to the Binomial Poisson Approximation to the Binomial If a binomial distribution has n > 50 and p < 0.1 then we can approximate it with a poison distribution where λ = np Assume we have a binomial distribution where n = 400 and p = 0.03 P(X = 5) = ( n C x ) p x q n x = ( 400 C 5 ) = We could approximate with a poisson where λ = np = 400(0.03) = 12 P(X = 5) = e ! = Dr. Nick Zammit (UofT) Topic 2 November 21, / 47 Approximations to the Binomial Normal Approximation to the Binomial If a binomial distribution has np 10 and nq 10 we can approximate it with a normal distribution where µ = np and σ = npq adjusted with the continuity correction The Continuity Correction To approximate binomial with normal make the adjustments: P(X x) P(X (x + 0.5)) P(X > x) P(X > (x 0.5)) P(X = x) P((x 0.5) X (x + 0.5)) We could approximate with a normal where µ = np = 12 and σ = (400)(0.03)(0.97) = P(X = 5) = P(4.5 X 5.5) = P( 2.20 Z 1.91) = Dr. Nick Zammit (UofT) = Topic 2 November 21, / 47

24 Supplementary References Dr. Nick Zammit (UofT) Topic 2 November 21, / 47