Course 003: Basic Econometrics

Size: px

Start display at page:

Download "Course 003: Basic Econometrics"

Geoffrey Morgan
5 years ago
Views:

1 Course 003: Basic Econometrics - Part 1 Delhi School of Economics, 2017 Page 0

2 Outline of the Part 1 Texts: Morris H. DeGroot and Mark J. Schervish, Probability and Statistics, fourth edition. Joseph Blitzstein and Jessica Hwang, Introduction to Probability. Topics Probability Theory: Probability basics: axioms, counting methods, independence, conditional probability. Random variables: distribution and density functions, marginal and conditional distributions, distributions of functions of random variables, moments of a random variable, properties of expectations. Special distributions : binomial, poisson, uniform, normal, gamma.. Convergence of random variables: laws of large numbers (LLN), central limit theorems (CLT) Statistical Inference: Estimation: maximum likelihood estimation, sufficient statistics, sampling distributions of estimators. Hypotheses Testing: simple and composite hypotheses, tests for differences in means, test size and power, uniformly most powerful tests. Page 1

3 Administrative Information Internal Assessment: 25% for Part 1 One midterm exam: 20%, September 25. Problem Sets and attendance : 5% must hand in during tutorials (not accepted later). Tutorials: Each group will have one tutorial per week and a lab session every fortnight. Punctuality is critical - coming in late disturbs the rest of the class Page 2

4 Why is this course useful? We (as economists, citizens, consumers, juries, exam-takers) are often faced with uncertainty. This may be caused by: randomness in the world -rain, sickness, future output incomplete information about a realized state of the world -Is a politician s promise sincere? Is a firm telling us the truth about a product? Has our opponent been dealt a better hand of cards? Is a prisoner guilty or innocent? Is a topic going to be on the exam? By putting structure on this uncertainty, we can arrive at decision rules: on technology, voting, conviction, study plans estimates of empirical relationships (wages and education, drugs and health, money supply and GDP ) tests of hypothesis: do we agree or disagree with the statement: smoking increase the chance of lung cancer, speed limits lower accident rates, NREGS reduces poverty. Probability theory puts structure on uncertain events. Statistics is about collecting and using data to make inferences about the population. Page 3

5 A motivating example: gender ratios Does the gender ratio in a population reflect discrimination? We visit a village and count the number of children below the age of 1. With no discrimination, the number of girls X Bin(n,.5). How should we decide that there is discrimination? Page 4

6 Gender ratios... Perhaps when the sample proportion, ˆp is below some threshold. But what threshold? The following table shows P( X n x) when p =.5 ˆp =.1 ˆp =.2 ˆp =.3 ˆp =.4 ˆp =.5 sample size Page 5

7 Gender ratios and sample size Binomial probabilities for n=20 and 100, p =.5 frequency frequency As the sample size increases, so does the threshold below which the null hypothesis is falsely rejected with a probability of less than.05 Page 6 binomial(20,5,.5)=.021, binomial(20,5,.5)=.058, binomial(100,42,.5)=.067 and binomial(100,41,.5)=.044

8 Basic definitions An experiment is any process whose outcome is not known in advance with certainty. These outcomes may be random or non-random, but we should be able to specify all of them and attach probabilities to them. A sample space S of an experiment is the set of all possible outcomes of an experiment. An event A is a subset of the sample space S. An event A has occurred A if the outcome is in Let A, B S be events. Then A B is the even that occurs if and only if at least one of A and B occurs, A B occurs of both A and B occur and A c (A complement) occurs if and only if A does not occur. When A 1, A 2, A of sample space S are disjoint sets the events A 1, A 2, A are said to be mutually exclusive. Page 7

9 Example: 3 tosses of a coin The experiment has 2 3 possible outcomes and we can define the sample space S = {s 1,..., s 8 } where s 1 = HHH, s 2 = HHT s 3 = HT H, s 4 = HT T, s 5 = T HH, s 6 = T HT, s 7 = T T H, s 8 = T T T Any subset of this sample space is an event. If we have a fair coin, each of the listed events are equally likely and we attach probability 1 8 to each of them. The events exactly one head and exactly two heads are mutually exclusive events. Page 8

10 The concept of probability A probability is a number attached to an event which expresses the likelihood of the event occurring. How are probabilities assigned to events? By thinking about all possible outcomes. If there are n of these, all equally likely, we can attach numbers n 1 to each of them. If an event contains k of these outcomes, we attach a probability n k to the event. This is the classical interpretation of probability. Alternatively, imagine the event as a possible outcome of an experiment. Its probability is the fraction of times it occurs when the experiment is repeated a large number of times. This is the frequency interpretation of probability In many cases events cannot be thought of in terms of repeated experiments or equally likely outcomes. We could base likelihoods in this case on what we believe about the world subjective probabilities. The subjective probability of an event A is a real number in the interval [0, 1] which reflects a subjective belief in the validity or occurrence of event A. Different people might attach different probabilities to the same events. Examples? We formalize this subjective interpretation by imposing certain consistency conditions on combinations of events. Page 9

11 Finite and infinite sample spaces Our goal is to assign probabilities to events in S. If S is finite, we can consider all possible subsets in S. If S has n elements, there are 2 n possible subsets. The set of all these subsets is called the power set of S. When S is infinite (such as the time we have to wait for a letter to arrive after an interview), it is not obvious how we should do this. Carefully defining which subsets can be assigned probabilities leads to the concept of a σ-algebra. Page 10

12 σ-algebras and Borel sets Definition: A σ-algebra on S is a collection F of subsets of S such that 1. F 2. If A F, then A c F 3. If A 1, A 2, F, then j=1 A j F In words: F contains and is closed under complements and countable unions. Intuition: If it makes sense to talk about the probability of an event happening, it makes sense to talk about it not happening, and if a set of events can occur, then at least one them can occur! Definition: A Borel σ-algebra B on R is defined to be the σ-algebra generated by all open intervals (a, b) with a, b R. A Borel set is a set in the Borel σ-algebra. Analogously, we define the Borel σ-algebra on R n to be the σ-algebra in R n generated by open boxes or rectangles in R n Page 11

13 Probability spaces and axioms Definition: A probability space is a triple (S, F, P), with S a sample space, F a σ-algebra on S and P, a probability measure defined on F Definition: A probability measure is a function on F, taking values between 0 and 1 such that: 1. P( ) = 0, P(S) = 1 2. P( ) j=1 A j) = P(A j ) if the A j are disjoint events. (countable additivity) j=1 We will usually use P(A j ) rather than P(A j ) Page 12

14 Probability measures... some useful results Use the probability axioms to derive the following useful results: 1. For each A S, P(A) = 1 P(A c ) 2. For A 1, A 2 S such that A 1 A 2, P(A 1 ) P(A 2 ) 3. For each A S, 0 P(A) 1 4. If A 1 and A 2 are subsets of S then P(A 1 A 2 ) = P(A 1 ) + P(A 2 ) P(A 1 A 2 ) 5. For a finite number of events, we have: n P( A i ) = i=1 n i=1 P(A i ) i<j P(A i A j ) + i<j<k Pr(A i A j A k )...( 1) n+1 P(A 1 A 2... A n ) Page 13

15 Examples 1. Consider two events A and B such that Pr(A) = 3 1 and Pr(B) = 2 1. Determine the value of P(BA c ) in each of the following cases: (a) A and B are disjoint (b) A B (c) Pr(AB) = Consider two events A and B, where P(A) =.4 and P(B) =.7. Determine the minimum and maximum values of Pr(AB) and the conditions under which they are obtained? 3. A point (x, y) is to be selected from the square containing all points (x, y), such that 0 x 1 and 0 y 1. Suppose that the probability that the point will belong to any specified subset of S is equal to the area of that subset. Find the following probabilities: (a) (x 1 2 )2 + (y 1 2 )2 1 4 (b) 2 1 < x + y < 2 3 (c) y < 1 x 2 (d) x = y answers: (1) 1/2, 1/6, 3/8 (2).1,.4 (3) 1-π/4, 3/4, 2/3, 0 Page 14

16 Counting methods: the multiplication rule A sample space containing n outcomes is called a simple sample space if the probability assigned to each of the outcomes s 1..., s n is n 1. Probability measures are easy to define in such spaces. If the event A contains exactly m outcomes, then P(A) = m n Counting the number of elements in an event and in the sample space can be laborious and sometimes complicated - we ll look at ways to make our job easier The multiplication rule: Sometimes we can think of an experiment being performed in stages, where the first stage has m possible outcomes and the second n outcomes. The total number of possible outcomes is then mn (e.g. a sandwich can have brown or white bread and then a tomato or cheese filling) B W c t Page 15

17 Samples, arrangements and combinations Suppose we are making k choices from n objects with replacement. There are n k possible outcomes How many arrangements of k objects from a total of n distinct objects can be had if we are sampling without replacement? The first object can be chosen in n different ways, leaving (n 1) objects so the second one can be picked in (n 1) different ways... The total number of permutations of n objects taken k at a time is then P n,k = n(n 1)... (n k + 1) and P n,n = n!. P n,k can alternatively be written as: (n k)! P n,k = n(n 1)..... (n k + 1) = n(n 1)..... (n k + 1) (n k)! = n! (n k)! How many different subsets of k elements can be chosen from a set of n distinct elements? Think of permutations as arising by first picking k elements and then organizing them in k! ways. Then the number of permutations is given by P n,k = k!c n,k, or C n,k = P n,k k! = n! k!(n k)! Page 16 This is called the binomial coefficient.

18 An application: the birthday problem You go to watch a cricket match with a friend. He would like to bet Rs. 100 that among the group of 23 people on the field (2 teams plus a referee) at least two people share a birthday Should you take the bet? What is the probability that out of a group of k, at least two share a birthday? the total number of possible birthdays is 365 k the number of different ways in which each of them has different birthdays is 365! (365 k)! (because the second person has only 364 days to choose from, etc.). The required probability is therefore p = 1 365! (365 k)!365 k It turns out that for k = 23 this number is.507, so you have a small expected monetary gain from the bet - if you don t like risk you probably should not take it Page 17

19 The multinomial coefficient Suppose we have k choices (jobs, modes of transport, methods of water filtration..) and are interested in number of ways that these can be chosen by n people such that for j = 1, 2,..., k j th group contains exactly n j elements. The n 1 elements for the first group can be chosen in ( n ) n 1 ways, the second group is chosen out of (n n 1 ) elements and this can be done in ( n n ) 1 n 2 ways...the total number of ways of dividing the n elements into k groups is therefore ( n n 1 )( n n1 n 2 )( n n1 n 2 n 3 ) (... nk 1 +n ) k n k 1 This can be simplified to n! n 1!n 2!...n k! This expression is known as the multinomial coefficient. Example An student organization of 1000 people is picking 4 office-bearers and 8 members for its managing council. The total number of ways of picking this groups is given by 1000! 4!8!988! Page 18

20 Independent Events Definition: Let A and B be two events in a sample space S. Then A and B are independent iff P(A B) = P(A)P(B). If A and B are not independent, A and B are said to be dependent. If events are independent, the occurrence of one provides no information on the likelihood of occurrence of the other. This may be because they are physically unrelated -tossing a coin and rolling a die, two different people falling sick with some non-infectious disease, but not necessarily: Example: The event A is getting an even number on a roll of a die. The event B is getting one of the first four numbers. The intersection of these two events is the event of rolling the number 2 or 4, which we know has probability 1 3. Are A and B independent? Yes because P(A)P(B) = = 1 3 But why? If A and B are independent, then A and B c are also independent as are A c and B c and A c and B. Why do you think this should be true? Page 19

21 Independent Events..examples and special cases 1. The experiment involves flipping two coins. A is the event that the coins match and B is the event that the first coins is heads. Are these events independent? In this case P(B) = P(A) = 1 2 ( {H,H} or {T,T}) and P(A B) = 1 4, so yes, the events are independent. 2. Suppose A and B are disjoint sets in S. Does it tell us anything about the independence of events A and B? 3. Remember that disjointness is a property of sets whereas independence is a property of the associated probability measure and the dependence of events will depend on the probability measure that is being used. Page 20

22 Independence of many events Definition: For n events, A 1, A 2, A independent. be independent, every finite subset of the events must be So pairwise independence is necessary but not sufficient. Examples: 1. One ticket is chosen at random from a box containing 4 lottery tickets with numbers 112, 121, 211, 222. The event A i is that a 1 occurs in the i th place of the chosen number. P(A i ) = 1 2 and P(A i A j ) = 1 4 so these 3 events are pairwise independent. But they are not independent since P(A 1 A 2 A 3 ) P(A 1 )P(A 2 )P(A 3 ) 2. Toss two fair dice. The sample space consists of all ordered pairs (i, j) i, j = 1, Define the following events : Page 21 A 1 : first die = {1, 2 or 3} A 2 : first die = {3, 4 or 5} A 3 : the sum of the faces equals 9 So P(A 1 ) = P(A 2 ) = 1 2 and P(A 3) = 1 9. P(A 1 A 2 A 3 ) = P(3, 6) = 36 1 = ( 1 2 )( 1 2 )( 1 9 ) = P(A 1)P(A 2 )P(A 3 ) but... P(A 1 A 3 ) = P(3, 6) = 36 1 P(A 1)P(A 3 ) = 18 1 So the events are not pairwise independent and so not independent.

23 Conditional probability Probabilities reflect our beliefs about the likelihood of uncertain events. Conditional probabilities reflect updated beliefs in the light of new evidence. All probabilities are conditional in that they reflect accumulated knowledge. Definition: If A and B are events with P(B) > 0, then the conditional probability of A, given B is defined as: P(A B) P(A B) = P(B) A is the event whose uncertainty we wish to update and B is the evidence we observe or the event we want to take as given. P(A) is called the prior probability of A and P(A B) is the posterior or revised probability of A 44 Introduction to Probability If B occurs, we eliminate B c and renormalize in the restricted sample space ( Fig 2.1-BH). A A A B B B FIGURE 2.1 Page 22 Pebble World intuition for P (A B). From left to right: (a) Events A and B are sub-

24 Conditioning on multiple events For any 3 events, A 1, A 2 and A 3 with positive probabilities, P(A 1 A 2 A 3 ) = P(A 1 )P(A 2 A 1 )P(A 3 A 1 A 2 ) We can generalize this to as many events as we like. For A 1, A 2,... A n : P(A 1 A 2... A n ) = P(A 1 )P(A 2 A 1 )P(A 3 A 1 A 2 )... P(A n A 1..., A n 1 ) Notice there are multiple forms of these expressions depending and which is most convenient to use depends on the problem at hand. Page 23

25 Bayes Rule Notice from the definition of conditional probability that the following expressions are equivalent: We can therefore write P(A B) = P(A B)P(B) = P(B A)P(A) P(A B) = P(B A)P(A) P(B) This expression for conditional probability of event A, given B is known as Bayes Rule. Page 24

26 The law of total probability Let A 1, A 2,... A k be a partition of the sample space S, with P(A i ) > 0 for all i. Then P(B) = k P(A i )P(B A i ) i=1 This is the law of total probability. Notice that the RHS is just P(BA 1 BA 2 BA k ) Using conditional probabilities sometimes makes it easy to solve a complicated problem. Example: You play a game in which your score takes integer values between 1 and 50 with equal probability. If your score the first time you play is equal to X, and you play until you score Y X, what is the probability that Y = 50? Solution: Let A i be the event X = x i and B is getting a 50 to end the game. P(X = x i ) = The probability of getting x i in the first round and 50 to end the game is given by the product, P(B A i )P(A i ). The required probability is the sum of these products over all possible values of i: P(Y = 50) = 50 x= x = 1 50 ( ) =.09 Page 25

27 Bayes Rule...examples C 1, C 2 and C 3 are plants producing 10, 50 and 40 per cent of a company s output. The percentage of defective pieces produced by each of these is 1, 3 and 4 respectively. Given that a randomly selected piece is defective, what is the probability that it is from the first plant? P(C 1 C) = P(C C 1))(P(C 1 ) P(C) = (.01)(.1) (.01)(.1) + (.03)(.5) + (.04)(.4) = 1 32 =.03 How do the prior and posterior probabilities of the event C 1 compare? What does this tell you about the difference between the priors and posteriors for the other events? Suppose that there is a new blood test to detect a virus. Only 1 in every thousand people in the population has the virus. The test is 98 per cent effective in detecting a disease in people who have it and gives a false positive for one per cent of disease free persons tested. What is the probability that the person actually has the disease given a positive test result? P(Disease Positive) = P(Positive Disease)P(Disease) P(Positive) = (.98)(.001) (.98)(.001) + (.01)(.999) =.089 So in spite of the test being very effective in catching the disease, we have a large number of false positives. Why? Page 26

28 Bayes Rule... priors, posteriors and politics To understand the relationship between prior and posterior probabilities a little better, consider the following example: A politician, on entering parliament, has a fairly good reputation. A citizen attaches a prior probability of 4 3 to his being honest. At the end of his tenure, there are many potholes on roads in the politician s constituency. While these do not leave the citizen with a favorable impression of the incumbent, it is possible that the unusually heavy rainfall over these years was responsible. Elections are coming up. How does the citizen update his prior on the moral standing of the politician? Let us compute the posterior probability of the politician s being honest, given the event that the roads are in bad condition: Suppose that the probability of bad roads is 1 3 if the politician is honest and is 2 3 if he/she is dishonest. The posterior probability of the politician being honest is now given by P(honest bad roads) = P(bad roads honest)p(honest) P(bad roads) = ( 1 3 )( 3 4 ) ( 1 3 )( 3 4 ) + ( 2 3 )( 1 4 ) = 3 5 What would the posterior be if the prior is equal to 1? What if it the prior is zero? What if the probability of bad roads was equal to 1 2 for both types of politicians? When are differences between priors and posteriors going to be large? Page 27

29 Conditioning matters: The Sally Clark case Sally Clark was a British solicitor who became the victim of a one of the great miscarriages of justice in modern British legal history Her first son died within a few weeks of his birth in 1996 and her second one died in similarly in 1998 after which she was arrested and tried for their murder. A well-known paediatrician Professor Sir Roy Meadow, who testified that the chance of two children from an affluent family suffering sudden infant death syndrome was 1 in 73 million, which was arrived by squaring 1 in 8500 for likelihood of a cot death in similar circumstance. Clark was convicted in November In 2001 the Royal Statistical Society issued a public statement expressing its concern at the misuse of statistics in the courts and arguing that there was no statistical basis for Meadow s claim In January 2003, she was released from prison having served more than three years of her sentence after it emerged that the prosecutor s pathologist had failed to disclose microbiological reports that suggested one of her sons had died of natural causes. Mistake: assumption of independence of outcomes, and confusing P(I E) P(E I) (I=innocent, E=evidence). If the prosecution had looked at P(I E), then the very large prior on innocence P(I) would have played a role. Page 28

30 . Conditional probability spaces When we condition on an event B, then B becomes our sample space and conditional probabilities replace our prior probabilities. In particular: P(S B) = 1, P( B) = 0 P( ) j=1 A j B) = P(A j B) if the A j are disjoint events in B j=1 P(A E B) = P(A B) + P(E B) P(A B E) We can condition on as many events as we like when applying Bayes rule and the law of total probability. If P(A B) and P(E B) are both greater than zero, Bayes Rule: P(A E, B) = P(E A,B)P(A B) P(E B) LOTP: P(E B) = k P(E A i, B)P(A i B) i=1 Conditional independence does not imply independence or vice-versa Page 29

Econ 325: Introduction to Empirical Economics

Econ 325: Introduction to Empirical Economics Lecture 2 Probability Copyright 2010 Pearson Education, Inc. Publishing as Prentice Hall Ch. 3-1 3.1 Definition Random Experiment a process leading to an uncertain