Discrete Distributions

Transcription

1 Discrete Distributions STA 281 Fall Introduction Previously we defined a random variable to be an experiment with numerical outcomes. Often different random variables are related in that they have the same sample space or the same form for the probabilities. For example, suppose X is a random variable with P(X=0)=0.3 and P(X=1)=0.7 while Y is a random variable with P(Y=0)=0.4 and P(Y=1)=0.6. These random variables are similar in that they have the same sample space S={0,1}. The random variables are different in that they assign different probabilities to the elements of the sample space. In fact, there are many random variables with the sample space S={0,1}. For every real number p in (0,1), we can define X such that P(X=1)=p and P(X=0)=1-p. These probabilities sum to 1 and are both non-negative because 0<p<1, thus the random variable satisfies the axioms of probability. When we have a set of similar random variables, we call this set a family of distributions. Typically families are indexed by a parameter such as p in the previous example. For each value of the parameter, we have a different random variable, but one that can be described by a common equation involving the parameter. The family we described in the previous example is called the Bernoulli family with parameter p. The parameter space is the possible values of the parameter. For the Bernoulli family, the parameter space is p (0,1). Bernoulli random variables are used in a variety of applications. The simplest example is flipping a fair coin. If we create a random variable X by assigning 1 to heads and 0 to tails, then X Bern(0.5) (since each outcome is equally likely, p must be 0.5). Bernoulli random variables are also used in voter polls. If Fred and Barney are running against each other in an election, there is some proportion p of people who will vote for Fred. If Fred is unpopular, then p may be small, if the race is close p may be near 0.5, or if Fred is winning by a landslide then p may be above 0.8. If we select one individual at random from the population and ask whether they are intending to vote for Fred, we get a yes or no answer. If we construct a random variable Y by assigning 1 to yes and 0 to no, then Y Bern(p). Any experiment resulting in two outcomes can be transformed to a Bernoulli by assigning one of the outcomes to 1 and the other outcome to 0. Because all Bernoulli random variables have a common set of possible values and a common form for the probabilities, we can solve the expectation and variance of a Bernoulli random variable in terms of the parameter p. Let X Bern(p). The expectation of the random variable is defined to be. The random variable X has two possible outcomes, 0 and 1, that occur with probabilities 1-p and p, so E[X]=p E[X 2 ]=p V[X]=p(1-p) So, for example, a Bern(0.7) random variable has mean 0.7 and variance Experiments Based on Bernoulli Distribution Bernoulli distributions form the basis for several important families of distributions. Complicated experiments can often be described by specific combinations of several Bernoulli random variables. 1

2 Since the Bernoulli distribution is simple, thinking of more complicated distributions in terms of Bernoulli random variables often makes them easier to understand. We will consider two experiments resulting from Bernoulli random variables. One experiment consists of drawing a sample of n Bernoulli random variables and counting how many ones appear in the sample. This experiment forms the basis of voter polling. Usually we don t just ask one person who they are going to vote for, but many. Gallup polls, for example, typically question around 1500 people. The count of the number of ones in the sample is used to make inferences about who will win the election. Another example is quality control, where we are interested in determining the proportion of products that are of acceptable quality. To assess the quality of the products, we test products as they come off the assembly line. Each product is classified as acceptable (1) or unacceptable (0). A count of acceptable items corresponds to a count of the number of ones. This description alone (take n Bernoulli s and count the number of successes) is not sufficient to specify the distribution. We will consider two specific variants. The first scenario, called a Binomial experiment, states that the n Bernoulli random variables are all independent and have the same probability of success p. If Y is the number of successes in this scenario, then Y has a Binomial distribution with parameters n and p (see section 2.1). When n is large and p is small, the Binomial distribution may be approximated by a Poisson distribution (see section 2.2). In contrast to a Binomial experiment, the n Bernoulli random variables may arise from sampling n items from a finite population divided into M successes and N-M failures. We have actually already considered this scenario. If Y counts the number of successes in this scenario, then Y has a Hypergeometric distribution with parameters N, M, and n (see section 2.3). Under some conditions, the Binomial distribution and the Hypergeometric distribution closely approximate each other (see section 2.4). The alternative experiment (as opposed to counting the number of successes in n trials) is to observe successive independent Bernoulli random variables, all with the same parameter p, and count the number of failures that occur before the r th success. Note in the previous experiment we fixed the number of trials, n, and counted the number of successes. We are doing the opposite here in that we have fixed the number of successes, r, and are counting the number of failures. We could equivalently count the number of trials, which is the number of failures plus the number of successes r. The number of failures before the r th success has a Negative Binomial distribution with parameters r and p (see section 2.5). 2.1 Binomial The Binomial distribution is one of the most commonly used distributions in statistics and, with the possible exception of the normal distribution, the distribution we will concentrate on most in this course. The binomial distribution describes the results of voter polls, clinical trials, and many other sampling procedures. A Binomial experiment has the following properties: 1. We observe a fixed, known number n of Bernoulli trials. 2. The Bernoulli trials are all independent with the same p. 3. The random variable of interest is the number of ones observed. The first requirement just says we know the maximum number of ones in advance. The second requirement is often satisfied in practice and proves mathematical convenience in describing a Binomial distribution. By assuming the Bernoulli s are independent, we may multiply probabilities together rather than using conditional probabilities. Assuming all the Bernoulli s have the same p further simplifies the calculations. 2

3 In a Binomial experiment we call our n Bernoulli trials X 1, X 2,,X n. We know that each X i Bern(p). Our random variable of interest is the number of ones among the Xi, so let us construct another random variable Y which is the sum of the n X i Such a random variable Y has a Binomial distribution with parameters n and p, written Y Bin(n,p). The Binomial random variable is just recording the number of ones in the sample, since each X i=1 increments Y while each X i=0 does nothing to Y. Recall that the distribution of a random variable is a list of the possible values combined with the probability of each value occurring. For simplicity, let us assume that n=4. If we observe 4 Bernoulli s and count the number of ones, we must observe 0, 1, 2, 3, or 4 ones. These are the possible values of Y. Calculating the probabilities of each of these outcomes requires looking at the individual Bernoulli probabilities. There are 4 Xi random variables, each with two possible values, 0 and 1. This results in 16 (2 4 ) possible combinations of the set (X1,X2,X3,X4) these are 0000, 0001, 0010,,1110,1111). Using the assumption that all the Xi are independent and have the same probability p, we may compute the probability of observing any particular set of Xi values. For example, the probability of observing X1=1, X2=0, X3=1, X4=1 is We may rewrite the intersection as a product because of the independence of the Xi. We may compute the probability for all of the sixteen combinations. In addition, for each combination we may compute the value of Y. For example, if X1=1, X2=0, X3=1, and X4=1, then Y= =3. The table shows the probabilities and Y for the sixteen combinations. X1 X2 X3 X4 Y Probability (1-p) p(1-p) p(1-p) p 2 (1-p) p(1-p) p 2 (1-p) p 2 (1-p) p 3 (1-p) p(1-p) p 2 (1-p) p 2 (1-p) p 3 (1-p) p 2 (1-p) p 3 (1-p) p 3 (1-p) p 4 Notice that all combinations with the same Y have the same probability. For example, all the rows where Y=2 have probability p 2 (1-p) 2. This is because all rows with Y=2 have 2 ones and 2 zeros, the ones occurring with probability p and the zeros occurring with probability (1-p). The sixteen combinations in the table are the sample space for the four Bernoulli s. To find P(Y=2), we may just 3

4 sum the probabilities of the outcomes corresponding to Y=2. There are six such combinations, each with probability p 2 (1-p) 2, so P(Y=2)=6p 2 (1-p) 2. We may also find P(Y=3) similarly. Each combination with Y=3 has 3 ones and 1 zero, so it occurs with probability p 3 (1-p). There are 4 rows with Y=3, so P(Y=3)=4p 3 (1-p). We could similarly determine the probabilities for the remaining values of Y. While n is always known in a Binomial experiment, we don t know it until we are told about it in the experiment, and so we want formulas for arbitrary values of n. If we count the number of ones for n Bernoulli trials, we will find the possible values are integers 0, 1, 2,, n. To find their probabilities, we could construct a table as with n=4, but if n is even moderately large the table will be too large to write conveniently. In principle such a table could be constructed and we could find the probabilities of each combination of the Xi. Each combination with y ones and n-y zeros will have probability p y (1-p) n-y (recall each one occurs with probability p and each zero occurs with probability 1-p). We must then multiply p y (1-p) n-y by the number of rows with Y=y to determine the probability. Each row with Y=y has y ones and n-y zeros. How many rows are there with y ones? This is the number of ways y ones may be placed in n slot, or n choose y. In general, if Y Bin(n,p), then To find the expectation of Y, we could look directly at the definition [ ]. This results in the complicated formula [ ] It is simpler to observe that Y is a linear combination of the Xi, and use the formulas for linear combinations to compute the mean and variance of Y. Recall that the mean and variance of each Xi are p and p(1-p) since each Xi is distributed Bern(p). Since we may derive [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] The expectation formula is intuitive. If we have 10 flips of a coin that is heads with probability 0.70, we expect about 10(0.7)=7 heads. Often we are more interested in the sample proportion than the actual number. For example, election polls usually report what percentage of those surveyed would vote for a particular candidate, not the actual count. If Y is the actual number of voters preferring the candidate, then the proportion that prefer the candidate is We may compute probabilities for sample proportions just as we would for counts. If we interview 1000 people, the probability the proportion will be is the same as the probability Y=712. The sample proportion is a linear transformation of Y, since. Using the formulas for the mean and variance of a linear transformation, we may derive 4

5 [ ] [ ] [ ] [ ] The expectation of is p, which indicates that, on average, the sample proportion is equal to the population proportion, one of several reasons the sample proportion is typically used to estimate p. The property that the estimator, on average, is equal to the actual value is called unbiasedness. The variance of decreases as the sample size n increases, because n is in the denominator of V[ ]. This indicates that as the sample size increases, the sample proportion becomes more precise (less variable). Combining the mean and variance of, we may observe that for large samples, the sample proportion is very likely to be close to the population proportion p. 2.2 Approximation of Binomial by Poisson Often, we are faced with a binomial experiment with a large n and a small p. For example, consider the number of pages in a book with typographical errors. After the book has been proofread many times, there is a small probability any given page will have an error, but a book contains many pages. A telephone company is often interested in how many people call their operator. They serve many customers, but in a given day each customer has only a small probability of calling the operator. In a football season, there is a small probability of a turnover on a given play, but a season consists of many plays. If we are interested in computing probabilities for the number of ones observed in a binomial experiment, we could use the binomial distribution. However, if n is large and p is small, the Poisson distribution is often used as a convenient approximation. As a rule of thumb, if n 100, p 0.05, and np 20, then a Bin(n,p) distribution may be accurately approximated with a Poisson distribution with parameter =np distribution, written Poi. As stated earlier, defining a random variable involves stating the possible values of the variable and giving their probabilities. If, then the possible values of X are all the nonnegative integers. Although the set of possible values differ between the Poisson and Binomial (the Binomial only has possible values 0,1,,n), the Poisson distribution places very little probability on values greater than n so the approximation is still reasonable. The possible outcomes have probabilities These probabilities sum to 1 since As an illustration, suppose we have a hotel with 100 rooms and that the probability any given room will ask for room service from 2AM to 6AM is Suppose we want to find exact and approximate probabilities that zero through five rooms will ask for room service. The exact distribution is Bin(n=100, p=0.05). The parameter values are sufficient to use a Poi( =np=5) distribution as an approximation. We computed probabilities according to both the Binomial and Poisson families. For example, let X be the number of rooms asking for room service. The exact probability (Binomial) that X=4 is 5

6 and the approximate probability (Poisson) is These are reasonably similar. Recall that the Poisson has an infinite number of possible values. We know that P(Y=101)=0, since there are not 101 rooms in the hotel. While the Poisson does give the event Y=101 nonzero probability, the approximate probability is While this value is not exactly 0, it is extremely close, so the approximation is still reasonable. For a fixed, the larger n is, the better the approximation. For X=0 through X=5, the approximate probabilities are the second row of the following table and the exact probabilities are the third row. Although not an exact approximation, the Poisson distribution is reasonable. The approximation improves as n increases, with np held constant. Suppose the hotel had 500 rooms and the probability any given room orders room service between 2AM and 6AM is The expected number of rooms ordering room service is also np=5, so a Poi(5) distribution could be used as an approximation. The exact probabilities are shown in the fourth row of the table. The approximation has improved accuracy for the larger n. If n is increased to 5000 and p decreased to (exact probabilities shown in the fifth row of the table), where np is also 5, the approximation is almost exact. Distribution Poi( =5) Bin(n=100, p=0.05) Bin(n=500, p=0.01) Bin(n=5000, p=0.001) If ), then E[X]= and V[X]=. These quantities approximate the Binomial expectation and variance. The expectations are the same, since =np. The variances are close, with the Poisson variance =np and the Binomial variance np(1-p). If p is small, the 1-p will be close to 1 so np(1-p) will be close to np. 2.3 Hypergeometric Distribution Suppose we have a population of N individuals consisting of M ones and N-M zeros (again, the ones and zeros may indicate any dichotomous (two valued) variable, such as gender). We sample n individuals from the population at random and let X be the number of ones in the sample. The random variable X then has a hypergeometric distribution with parameters N, M, and n, written HyperGeo(N,M,n). While we still have a sample of n individuals from a population and record the number of ones in the sample (same as the Binomial), here our sampling scheme creates dependencies among the Bernoulli trials. We could consider sampling n individuals at random to occur by picking a single individual at random from the N individuals, then picking a second individual from the N-1 individuals remaining, and so on. The Bernoulli s are dependent because whether a one or a zero is chosen on the second pick depends on whether a one or zero was chosen on the first pick. For example, suppose a class consists of 30 men and 10 women, and we select 5 students at random. The probability we select a man on the first pick is 30/40. If a man is chosen with the first pick, the probability a man is chosen on the second pick 6

7 is 29/39. In contrast, if a woman is chosen with the first pick, the probability a man is chosen with the second pick is 30/39. Since the probability of choosing a man with the second pick depends on the gender of the first person picked, the Bernoulli trials are dependent and thus the Binomial distribution does not apply. Although this is not a Binomial experiment, we have already computed the probabilities for this scenario. We want to find the probability of randomly selecting x ones from a population of M ones and N-M zeros. The probability X=x is The possible values of this distribution require some thought to determine. Clearly x must be an integer, and must be at least 0 and at most n. However, depending on the number of ones and zeros in the population, some values may not be possible. For example, if the population consists entirely of zeros, then the only possible value is x=0. The largest value x may be is n, provided there are enough ones in the population that n ones may be selected. If M is less than n, so there are fewer ones available that the number of individuals we are selecting, then we may observe at most M ones. The largest possible value of x is therefore min(n,m). Similarly, if the number of zeros in the population is fewer than the number of individuals we sample, then we must observe at least n-(n-m)=n-n+m ones. The possible values of x are therefore any integer between max(0,n-n+m) and min(n,m). Through some algebra, it is possible to determine the mean and variance of the hypergeometric distribution. If X HyperGeo(N,M,n), then [ ] [ ] 2.4 Comparison of Binomial and Hypergeometric Distribution Recall we said that for a Hypergeometric distribution, the Bernoulli trials are dependent. We said that in a population of 30 men and 10 women, the probability of selecting a man on the second pick depended on the gender of the first pick. We said that if a man was picked first, there was a 29/39 probability of selecting a man second. If a woman was picked first, there was a 30/39 probability of selecting a man second. The difference between 29/39 and 30/39 is not that great. There is dependence, but is it enough to make a difference? In fact, the answer is no, that the dependence between the picks is sufficiently small that we may ignore it. As a general rule, the larger the population, the easier it is to ignore the dependence. If, for example, class consisted of 4000 individuals with 1000 women, the difference between 3000/3999 and 2999/3999 is extremely small ( versus ). For large populations, the difference between the probabilities is so small that often we completely ignore the dependence and use the binomial distribution. In voter polling, often there are millions of voters. Although technically the correct distribution is Hypergeometric, the Binomial is always used to compute margins of error. Suppose the class consists of 25 individuals with 5 ones and you select 5 individuals. Let X be the number of ones observed in the sample. The possible values of X are the integers zero through five. The Hypergeometric distribution is the exact distribution, since we are drawing without replacement 7

8 from a finite population. We may find the hypergeometric probability directly from the definition, such as The full set of probabilities is shown in the second row of the table. We could approximate these probabilities by a Binomial distribution. We are sampling n=5 individuals from a population that contains p=5/25=0.2 proportion of ones. The binomial probabilities are shown in the fifth row of the table. The values are not very similar, although they are not vastly different, either. The accuracy of the binomial distribution increases as N increases (with the proportions remaining equal). Suppose the class consisted of 250 individuals with 50 ones (p=0.2) and we sample 5 individuals. The hypergeometric probabilities are shown in the third row of the table and match the binomial probabilities fairly close. As N is increased further, to 2500, also with M=500 so p=0.2, the probabilities are almost exactly equal. Distribution HyperGeo(N=25,M=5,n=5) HyperGeo(N=250,M=50,n=5) HyperGeo(N=2500,M=500,n=5) Bin(n=5,p=0.2) As a rule of thumb, HyperGeo(N,M,n) may be approximated by a Bin(n,p=M/N) if (n/n) 0.05, so we are sampling no more than 5% of the population. When this approximation holds, the mean and variance of the approximating Binomial are close to the mean and variance of the original Hypergeometric distribution. Recall the expectation of a Binomial distribution is np, which for the approximation is nm/n. This is exactly equal to the expectation of the Hypergeometric. The variance of the binomial is np(1-p), which for the approximation is Compare this to the exact variance of the Hypergeometric [ ] The only difference between the exact variance and the approximate is the term (N-n)/(N-1). However, remember we are using this approximation for n/n 0.05, which would make the (N-n)/(N-1) term approximately Negative Binomial Distribution So far we have concentrated on collecting a sample of n Bernoulli random variables and counting the number of ones in the sample. Another experiment we could perform is to sequentially observe Bernoulli random variables until we observe the first one, or the first r ones. For simplicity, we first describe observing Bernoulli s until we observe the first one. After observing the first one, we let a random variable Y be the number of zeros observed. For example, suppose the first three Bernoulli trials are X1=0, X2=0, and X3=1. The first one appeared on the third trial, and we observed Y=2 zeros before the first one. The possible values of Y are any nonnegative integers 0,1,2,. It could take an arbitrarily long time to observe the first one. Usually we will observe only the first one fairly quickly, so large values of Y have small probabilities, but they are possible. 8

9 To complete the description of the distribution, we also require the probabilities for each of the possible outcomes. If the Bernoulli trials we observe are independent and have the same probability p, this is relatively easy to calculate. To find P(Y=0), we must realize that the event Y=0 corresponds to observing no zeros before the first one. So Y=0 is the same event as X1=1, which we know occurs with probability p. The event Y=1 occurs if we observe one zero before the first one, so X1=0 and X2=1. This occurs with probability p(1-p). In general, the event Y=y corresponds to observing y zeros before the first one, so X1 through Xy are all 0 while Xy+1=1. This occurs with probability p(1-p) y. This distribution, the number of zeros until the first one, is called the Geometric distribution. More generally, instead of looking at the number of zeros observed before the first one, we may look at the number of zeros before the second one, or third one, or r th one. This is called the Negative Binomial distribution. The Geometric distribution is a Negative Binomial distribution with r=1. As with the geometric, if Y is the number of zeros before the r th one, the possible values are all nonnegative integers. We might find r ones in the first r trials, and therefore observe no zeros. Similarly, it could take an arbitrarily large number of Bernoulli s before the r th one. Suppose we are waiting for the 5 th one. What is the probability we observe exactly 7 zeros before the 5 th one? We calculate this probability by noticing that the event The 5 th one occurs after exactly 7 zeros requires that we observe 12 Bernoulli trials, with 5 ones and 7 zeros, and that the 12 th Bernoulli trial must have been a one, since we are waiting for the 5 th one to appear. Therefore, the first 11 Bernoulli s consist of 7 zeros and 4 ones, and the 12 th Bernoulli is a one. There are no restrictions on the order of ones and zeros for the first 11 Bernoulli s, so this occurs with probability This quantity must be multiplied by the probability that the 12 th Bernoulli is a one, which is p, so the actual probability is In general, we want the probability of observing y zeros before we observe the r th one. For the event Y=y to occur, we must observe y+r Bernoulli s, the first y+r-1 Bernoulli s consisting of y zeros and r-1 ones, and then observing a one on the next Bernoulli trial. The probability is The mean and variance of the negative binomial distribution are E[Y]=r(1-p)/p and V[Y]=r(1-p)/p 2. 3 Recognizing Distributions This handout discusses five distributions (Bernoulli, Binomial, Hypergeometric, Poisson, and Negative Binomial). One skill you should have is the ability to look at a word problem and determine which distribution is appropriate. After that, of course, you need to be able to apply formulas to find probabilities, means, and variances. The Bernoulli distribution is the building block of all other distributions. It should be the simplest to recognize, since it has just two outcomes, 0 and 1, which occur with probabilities p and 1-p. Bernoulli s many be combined in two ways in this course. Either we sample a group of n Bernoulli s from a population and count the number of ones in the sample, or we observe a sequence of Bernoulli s until we observe r ones and then count the number of zeros before the r th one occurred. The Binomial, 9

10 Hypergeometric, and Poisson distributions describe counting the number of ones while the Negative Binomial distribution describes counting how many zeros until the r th one. If we are sampling n independent Bernoulli trials, all with the same probability p of success, then you should use a Bin(n,p) distribution. If you are randomly sampling n individuals from a population with M successes and N-M failures, use a HyperGeo(N,M,n) distribution. If you are doing neither, consult your friendly local statistician. There are two approximations we consider in this course. If the exact distribution is Bin(n,p) but n is large and p is small, the Poi( =np) distribution is an adequate approximation. If the exact distribution is HyperGeo(N,M,n) but n/n 0.05, then the Bin(n,p=M/N) distribution is an adequate approximation. 10