Introduction to Statistical Data Analysis Lecture 3: Probability Distributions

Transcription

1 Introduction to Statistical Data Analysis Lecture 3: Probability Distributions James V. Lambers Department of Mathematics The University of Southern Mississippi James V. Lambers Statistical Data Analysis 1 / 56

2 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation Introduction Now that we know how to compute probabilities of events, we can study the behavior of the probability across all possible outcomes of an experiment that is, the distribution of the probability across the sample space. Our understanding of the probability distribution will eventually allow us to make inferences from the data from which the distribution arises. James V. Lambers Statistical Data Analysis 2 / 56

3 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation Random Variables A random variable, usually denoted by a capital letter such as X, is an outcome of an experiment that has a numerical value. The value itself is usually denoted by the lower-case version of the letter used to denote the variable itself; that is, a random variable X takes on numerical values that are denoted by x. Random variables can either be continuous or discrete. A continuous random variable can assume a value equal to any real number within some interval, whereas a discrete random variable can only assume selected numerical values, such as, for example, nonnegative integers. We will study random variables of both kinds. James V. Lambers Statistical Data Analysis 3 / 56

4 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation Discrete Probability Distributions A discrete probability distribution is a listing of all possible values of a discrete random variable, along with the probability of each value being assumed by the variable. James V. Lambers Statistical Data Analysis 4 / 56

5 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation Example Let X be a discrete random variable whose outcomes correspond to where one finishes in a race: first, second, third, etc. If there are 10 runners in the race, then X can assume as a value any positive integer between 1 and 10. James V. Lambers Statistical Data Analysis 5 / 56

6 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation The Distribution The probability distribution might look like the following: x P(X = x) Note that the notation P(X = x) is used to refer to the probability that the random variable X assumes the value x. James V. Lambers Statistical Data Analysis 6 / 56

7 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation Rules for Discrete Distributions A discrete probability distribution must follow these rules: Each outcome must be mutually exclusive of the others; that is, we cannot have X assume two values simultaneously as a the result of an experiment. For each outcome x, we must have 0 P(X = x) 1. If the distribution has n possible outcomes x 1, x 2,..., x n, then we must have n P(X = x i ) = 1. i=1 James V. Lambers Statistical Data Analysis 7 / 56

8 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation Mean For a given probability distribution, it is very helpful to know the most likely, or expected, value that the variable will assume. This can be obtained by computing a weighted mean of the outcomes, where the probabilities serve as the weights. We therefore define the mean, or expected value, of the discrete random variable X by n E[X ] = µ = x i P(X = x i ). i=1 James V. Lambers Statistical Data Analysis 8 / 56

9 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation Example Consider a raffle, in which each ticket costs $5. There is one grand prize of $100, two first prizes of $50 each, and four second prizes of $25 each. If 200 tickets are sold, then the probability of winning the grand prize is 1/200 = 0.005, while the probabilities of winning first prize and second prize are 2/200 = 0.01 and 4/200 = 0.02, respectively. Then, the expected amount of winnings is E[X ] = 100(0.005) + 50(0.01) + 25(0.02) + 0(0.965) = 1.5. James V. Lambers Statistical Data Analysis 9 / 56

10 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation Interpretation That is, a ticket holder can expect to win, on average, $1.50. However, we must account for the cost of the ticket, which applies to all participants; therefore, the expected net winnings is $3.50. Since the expected amount is negative, the raffle is not fair to the ticket holders; if the expected value was zero, then the raffle would be considered a fair game. James V. Lambers Statistical Data Analysis 10 / 56

11 Random Variables Discrete Probability Distributions Rules for Discrete Distributions Mean Variance and Standard Deviation Variance and Standard Deviation Using the mean of X, we can then characterize the dispersion of the outcomes by defining the variance of X as follows: σ 2 = n (x i µ) 2 P(X = x i ). i=1 An equivalent formula, in terms of expected values, is σ 2 = E[X 2 ] E[X ] 2. Note that in the first term, the values of X are squared, and then they are multiplied by the probabilities and summed, whereas in the second term, the expected value is computed first, and then squared. James V. Lambers Statistical Data Analysis 11 / 56

12 The uniform distribution U{a, b} is the probability distribution for a random variable X with domain {a, a + 1,..., b} in which each value in the domain of X is equally likely to be observed It follows that the probability mass function for this distribution is P(X = k) = 1, n = b a + 1, k {a, a + 1,..., b} n James V. Lambers Statistical Data Analysis 12 / 56

13 Mean and Variance Using the above definitions of the mean and variance of a discrete random variable, it can be shown that E[X ] = a + b 2, σ2 = (b a + 1) If a random variable X has the distribution U{a, b}, we write X U{a, b} We will use similar notation with other probability distributions, in order to indicate that a given random variable has a particular distribution James V. Lambers Statistical Data Analysis 13 / 56

14 Binomial Experiments The The Mean and Standard Deviation Binomial Experiments Suppose that an experiment is performed n times, and it can have only two outcomes, that are classified as success and failure. Each of these individual experiments is referred to as a trial. Furthermore, suppose that each trial is independent of the others, and that the probability of a trial being successful is p, where 0 < p < 1 (and therefore, the probability of failure is q = 1 p). These trials are called Bernoulli trials. James V. Lambers Statistical Data Analysis 14 / 56

15 Binomial Experiments The The Mean and Standard Deviation Examples Examples of Bernoulli trials are: Testing for defective parts, in which n is the number of parts to be checked, p is the probability that a part is not defective, and k is the number of parts that are not defective. Observing the number of correct responses on exam, in which n is the total number of questions, p is the probability of getting the correct answer on a single question, and k is the number of correct responses. Counting number of households with an internet connection, in which n is the number of households, p is the probability of a single household having an internet connection, and k is the number of households that have an internet connection. James V. Lambers Statistical Data Analysis 15 / 56

16 Binomial Experiments The The Mean and Standard Deviation The The binomial distribution B(n, p) is the probability distribution for the discrete random variable X whose value is the number of successes, denoted by k, in n Bernoulli trials, with probability of success p for each trial. Given a value for k, 0 k n, what is P(X = k), the probability that X is equal to k? First, we note that because the trials are independent, the probability of success (or failure) in consecutive trials can be obtained simply by multiplying the probabilities of the outcomes of the individual trials. It follows that the probability of k successes, followed by n k failures, is p k (1 p) n k. James V. Lambers Statistical Data Analysis 16 / 56

17 Binomial Experiments The The Mean and Standard Deviation Probability Mass Function However, to determine the probability that any k of the n trials are successful, we have to consider all possible ways to choose k trials out of the n to be successful. That is, we must multiply the above expression by n C k. We conclude that the probability mass function for the binomial distribution is P(X = k) = n C k p k (1 p) n k = n! k!(n k)! pk (1 p) n k. Using properties of the binomial coefficients, it can be verified that the sum of all of these probabilities, for k = 0, 1, 2,..., n, is equal to 1. James V. Lambers Statistical Data Analysis 17 / 56

18 Binomial Experiments The The Mean and Standard Deviation Examples The binomial distribution, for various values of n and p James V. Lambers Statistical Data Analysis 18 / 56

19 Binomial Experiments The The Mean and Standard Deviation Behavior of the Distribution Note that the binomial distribution is symmetric if p = 0.5, in which case the probability mass function simplifies to P(X = k) = n C k 2 n. Otherwise, the distribution skews to the left if p < 0.5, because there is a greater probability of more failures, and skews to the right if p > 0.5, since there is a greater probability of more successes. James V. Lambers Statistical Data Analysis 19 / 56

20 Binomial Experiments The The Mean and Standard Deviation The Mean and Standard Deviation Using the definition of expected value, and properties of binomial coefficients, it can be shown by direct computation, and a lot of algebraic manipulation, that if X is a discrete random variable with a binomial distribution corresponding to n trials and probability of success p, then E[X ] = µ = np. It can also be shown that the standard deviation is given by σ = np(1 p). James V. Lambers Statistical Data Analysis 20 / 56

21 Binomial Experiments The The Mean and Standard Deviation in R In R, the function dbinom can be used to compute probabilities from a binomial distribution. Its first argument is a value, or vector of values, of k (number of successes). The second argument is n, the number of trials, and the third argument is p, the probability of success. An example of its usage is: > dbinom(c(0,1,2,3,4),4,0.5) [1] The output lists P(X = k), for k = 0, 1, 2, 3, 4, with p = 0.5 and n = 4. James V. Lambers Statistical Data Analysis 21 / 56

22 The binomial distribution involves sampling with replacement, because each trial is independent of the other trials. By contrast, the hypergeometric distribution is based on sampling without replacement. Suppose that n trials are to be performed, but the outcomes of these trials are drawn from a set of N + M outcomes, of which N are successes and M are failures. The hypergeometric distribution describes the probability that k of the n trials are successes. James V. Lambers Statistical Data Analysis 22 / 56

23 Example A situation that would call for the hypergeometric distribution is the following: Suppose that you have 100 lightbulbs, and you know that 10 of them are defective. If you need 20 lightbulbs and you start taking them from the collection of 100, what is the probability that at most 2 of the chosen lightbulbs are defective? James V. Lambers Statistical Data Analysis 23 / 56

24 Probability Mass Function To compute the probability of k successes out of n trials, we need to count the number of ways to choose k objects (the successes) out of a set of N, and then choose n k objects (the failures) out of a set of M. This is divided by the number of ways to choose n objects from a set of N + M, to obtain the probability mass function ( ) ( ) N M k P(X = k) = ( N + M n n k ). James V. Lambers Statistical Data Analysis 24 / 56

25 Mean and Variance It can be shown that if X is a random variable that follows the hypergeometric distribution, and p = N/(N + M) is the probability of success in a single trial, then the mean and variance are given by E[X ] = np, Var(X ) = np(1 p) ( 1 n 1 N + M 1 It is interesting to note as N + M in such a way that p remains fixed, the variance converges to that of the binomial distribution, which makes sense because as the number of outcomes increases, the distinction between sampling with replacement and sampling without replacement diminishes. ). James V. Lambers Statistical Data Analysis 25 / 56

26 Example Continuing the previous example of sampling lightbulbs, the probability of at most two defective lightbulbs (failures), or at least 18 successes, is P(X 18) = P(X = 18) + P(X = 19) + P(X = 20) ) ) ) ( 90 ) ( 10 = 18 2 ( ) = = ( 90 ) ( ( ) + ( 90 ) ( ( ) James V. Lambers Statistical Data Analysis 26 / 56

27 Poisson Processes The Approximating the Poisson Processes In contrast to Bernoulli trials, in which an experiment consists of a fixed number of trials and the number of successful trials is counted, a Poisson process is an experiment that counts the number of occurrences of a certain outcome over a certain period of time, area, or other domain-defining quantity. In addition, a Poisson process has these defining characteristics: The mean number of occurrences must be the same for each interval of measurement, and The number of occurrences within an interval must be independent of those in any other interval. James V. Lambers Statistical Data Analysis 27 / 56

28 Poisson Processes The Approximating the Examples of Poisson processes Car accidents within a particular area Requests for documents from a web server Calls received by a call center Customers entering a queue James V. Lambers Statistical Data Analysis 28 / 56

29 Poisson Processes The Approximating the The Suppose a Poisson process has a mean of µ. Then, the probability distribution of the process P(µ) is described by the probability mass function P(X = k) = µk e µ, k = 0, 1, 2,.... k! It can be shown that the variance is actually equal to the mean, and therefore the standard deviation is µ. James V. Lambers Statistical Data Analysis 29 / 56

30 Poisson Processes The Approximating the Various s The Poisson distribution P(µ), for various values of µ James V. Lambers Statistical Data Analysis 30 / 56

31 Poisson Processes The Approximating the Example Suppose that a tire manufacturing plant determines that on average, 0.5 defective tires are produced per hour. Then, if X is the random variable for the number of defective tires per hour, the probability that 1 defective tire is produced within the next hour is given by P(X = 1) = (0.5)1 e 0.5 1! = e = James V. Lambers Statistical Data Analysis 31 / 56

32 Poisson Processes The Approximating the Example, cont d To determine the probability that at most 3 defective tires will be produced during the next day, where a work day is defined to be 8 hours, we use the fact that the mean is the same for each interval of measurement to determine that on average, 4 defective tires will be produced per day. Then, the probability of at most 3 defective tires is given by 3 P(X = k) = 40 e 4 0! k= e 4 1! + 42 e 4 2! + 43 e 4 3! = James V. Lambers Statistical Data Analysis 32 / 56

33 Poisson Processes The Approximating the in R The R function dpois gives the probability for given values of k (first argument) with a specified mean µ (second argument). To easily compute cumulative probabilities use the ppois function. n P(X = k), k=0 The first argument is n, the highest number of desired outcomes, and the second argument is the mean µ. James V. Lambers Statistical Data Analysis 33 / 56

34 Poisson Processes The Approximating the in R, cont d The following output gives two ways to perform the computation from the preceding example: > sum(dpois(c(0,1,2,3),4)) [1] > ppois(3,4) [1] James V. Lambers Statistical Data Analysis 34 / 56

35 Poisson Processes The Approximating the Approximating the The Poisson distribution can be used to approximate the binomial distribution. This is useful because evaluating the probability mass function P(X = k) = µk e µ is easier than evaluating the binomial distribution function P(X = k) = k! n! k!(n k)! pk (1 p) n k. James V. Lambers Statistical Data Analysis 35 / 56

36 Poisson Processes The Approximating the Assumptions The mean np of the binomial distribution can be substituted for the value of µ, the mean of the Poisson distribution. This approximation works well provided that the number of trials n is at least 20, and the probability of success p is quite small, at most Otherwise, the Poisson distribution is not a good fit for the binomial distribution curve. James V. Lambers Statistical Data Analysis 36 / 56

37 Poisson Processes The Approximating the Assumptions are Important! Note that in the left plot, the parameters n and p do not satisfy the conditions n 20, p 0.05, and therefore the two distributions do not agree very well. In the right plot, the parameters do satisfy the conditions (barely), and the fit is much better. Approximation of the binomial distribution (blue circles) by the Poisson distribution (red crosses) for n = 15, p = 0.1 (left plot) and n = 20, p = 0.05 (right plot). James V. Lambers Statistical Data Analysis 37 / 56

38 Continuous Exponential Distribution Normal Distribution Continuous Probability Distribution Recall that a continuous random variable is a random variable X whose domain is an interval D = [a, b], which is a subset of R, the set of real numbers A continuous probability distribution is a function f : D [0, 1] whose value at x D is the probability P(X = x) The function f (x) is the probability density function of X. By analogy with the requirement that the sum of all probabilities in a discrete probability distribution must equal one, a probability density function for a continuous random variable X must satisfy b a f (x) dx = 1, where the interval [a, b] is the domain of X James V. Lambers Statistical Data Analysis 38 / 56

39 Continuous Exponential Distribution Normal Distribution Mean and Variance The mean, or expected value, of a continuous random variable X is defined by E[X ] = b a xf (x) dx Then, we can define the variance in the same way as for a discrete random variable: Var[X ] = E[X 2 ] E[X ] 2 James V. Lambers Statistical Data Analysis 39 / 56

40 Continuous Exponential Distribution Normal Distribution Continuous The continuous uniform distribution U(a, b) is the probability distribution for a random variable X with domain [a, b] in which all subintervals of [a, b] of the same width are equally likely to be observed It follows that the probability density function for this distribution is f (x) = 1, x [a, b] b a Using the above definitions of the mean and variance of a continuous random variable, it can be shown that E[X ] = a + b 2, (b σ2 a)2 = 12 James V. Lambers Statistical Data Analysis 40 / 56

41 Continuous Exponential Distribution Normal Distribution Continuous in R The R function dunif gives the probability of observing within a subinterval of width 1 centered at x (first argument) on a specified interval [a, b] (second and third arguments). It simply returns 1/(b a) if a x b, and 0 otherwise. To easily obtain cumulative probabilities, use the punif function. The first argument is c, the largest desired outcome, and the second and third arguments are the endpoints a and b, respectively, of the domain of X Finally, given a probability p, the function qunif(p,a,b) returns the value of x (that is, the quantile) such that P(X x) = p. It can easily be determined that x = p(b a) + a James V. Lambers Statistical Data Analysis 41 / 56

42 Continuous Exponential Distribution Normal Distribution Exponential Distribution The exponential distribution Exp(λ) is a continuous distribution that describes the time between events in a Poisson process Its parameter λ is a nonnegative real number that is called the rate parameter It refers to the number of events per unit of time that are expected to occur. James V. Lambers Statistical Data Analysis 42 / 56

43 Continuous Exponential Distribution Normal Distribution The Particulars The probability density function for a continuous random variable X Exp(λ) is f (x) = λe λx, and its mean and variance are E[X ] = 1 λ, Var[X ] = 1 λ 2 For example, suppose an operator at a call center receives, on average, two calls per hour. Then the time between calls is a random variable X Exp(2), and its mean is E[X ] = 1/2. That is, the operator can expect to receive a call every half hour James V. Lambers Statistical Data Analysis 43 / 56

44 Continuous Exponential Distribution Normal Distribution Example Suppose that you are renting a car late at night, and there is only one customer service representative working at the counter. On average, he can assist 10 customers per hour, or one customer every 6 minutes. This corresponds to a rate parameter of 1/6 customers per minute. If he just started helping a customer, and you are at the front of the line, what is the probability that you will get to the counter within the next 5 minutes? That probability is P(X 5) = e x/6 dx = 1 e 5/6 = That is, you have a 57% chance of being waited on within 5 minutes. James V. Lambers Statistical Data Analysis 44 / 56

45 Continuous Exponential Distribution Normal Distribution Normal Distribution The normal distribution is a probability distribution that is followed by continuous random variables, that can assume any real value within some interval. A normal distribution has two parameters, its mean µ and its standard deviation σ; often, N (µ, σ) is used to refer to a specific normal distribution. Its mean, median and mode are all the same, and equal to µ. James V. Lambers Statistical Data Analysis 45 / 56

46 Continuous Exponential Distribution Normal Distribution Characteristics The distribution is bell-shaped, and is symmetric around the mean. In view of the essential properties of probability, the area under the entire bell-shaped normal distribution curve must be equal to 1. The probability is always strictly positive; it can never be zero, though the probability approaches zero for values of the variable that are far from the mean. The probability density function is P(X = x) = 1 σ 2 2π e (x µ) /(2σ 2). James V. Lambers Statistical Data Analysis 46 / 56

47 Continuous Exponential Distribution Normal Distribution Normal Distribution in R This function can be evaluated in R using its function dnorm; for example, dnorm(1,0.5,2) computes P(X = 1) for the normal distribution with mean µ = 0.5 and standard deviation σ = 2. If the third argument is omitted, then σ is assumed to be 1; if the second argument is omitted as well, then µ is assumed to be 0. This corresponds to the notion of the standard normal distribution, that has mean 0 and standard deviation 1. James V. Lambers Statistical Data Analysis 47 / 56

48 Continuous Exponential Distribution Normal Distribution The Standard Normal Distribution The standard normal distribution, with mean 0 and standard deviation 1 James V. Lambers Statistical Data Analysis 48 / 56

49 Continuous Exponential Distribution Normal Distribution Calculating Probabilities Suppose we wish to determine P(X x 0 ), which happens to the area of the region bounded by the normal distribution curve, the x-axis, and the vertical line x = x 0. As such, this probability would be given by P(X x 0 ) = x0 P(X = x) dx = 1 σ 2π x0 e (x µ)2 /(2σ 2) dx, but this integral cannot be evaluated using analytical techniques from calculus. It must instead be evaluated numerically, which is cumbersome. James V. Lambers Statistical Data Analysis 49 / 56

50 Continuous Exponential Distribution Normal Distribution More R Functions In R, we can use the pnorm function; for example, pnorm(1) computes P(X 1) for the normal distribution with µ = 0 and σ = 1. More generally, to compute the probability P(X x 0 ): pnorm(x0,m,s) where m is the mean and s is the standard deviation To find the quantile x 0 such that P(X x 0 ) = a: x0=qnorm(a,m,s) As with dnorm, the default values of m and s are 0 and 1, respectively James V. Lambers Statistical Data Analysis 50 / 56

51 Continuous Exponential Distribution Normal Distribution Tables and z-scores Tables are often used to evaluate normal distribution probabilities. Such tables use the standard normal distribution N (0, 1); therefore, if a different distribution is being used, a conversion to the standard distribution must be performed first. This involves computing the z-score, z = x µ σ. If x is a value of the normal distribution N (µ, σ), then z is the corresponding value in N (0, 1); more precisely, it is the number of standard deviations between x and µ. James V. Lambers Statistical Data Analysis 51 / 56

52 Continuous Exponential Distribution Normal Distribution Using Symmetry We can now describe how to compute various probabilities using normal distribution tables. In the following, we assume that z 0 is the z-score for x 0. P(X x 0 ): Obtain P(Z z 0 ) from a standard normal distribution table, or by using pnorm P(X > x 0 ) = 1 P(X x 0 ), because the events X > x 0 and X x 0 are complementary. That is, they are mutually exclusive and exhaustive, so their probabilities must sum to 1. P(X µ x 0 ) = 1 P(X µ + x 0 ), by the symmetry of the normal distribution. P(X > µ x 0 ) = P(X µ + x 0 ), again by symmetry. P(x 1 X x 2 ) = P(X x 2 ) P(X x 1 ). James V. Lambers Statistical Data Analysis 52 / 56

53 Continuous Exponential Distribution Normal Distribution The Empirical Rule, Revisited The empirical rule, introduced previously, can be used to estimate normal distribution probabilities. While it is approximately true for any bell-shaped, symmetric distribution, it is exact for any normal distribution. In fact, the rule is derived from the behavior of the normal distribution. Expressed in terms of probabilities, the empirical rule states that P( 1 Z 1) 0.68, P( 2 Z 2) 0.95, P( 3 Z 3) James V. Lambers Statistical Data Analysis 53 / 56

54 Continuous Exponential Distribution Normal Distribution Approximating the Like the Poisson distribution, the normal distribution can be used to approximate the binomial distribution, as long as the number of trials n and the probability of success p satisfy np 5, n(1 p) 5. James V. Lambers Statistical Data Analysis 54 / 56

55 Continuous Exponential Distribution Normal Distribution Un-discretization For computing probabilities, it is best to use the midpoints of the discrete values of the number of successes. For example, to approximate P(X 5), where X is a discrete random variable with a binomial distribution, one should work with a continuous random variable Y with a normal distribution N (np, np(1 p)) and compute P(Y 4.5), rather than P(Y 5). This is due to the change from a discrete random variable to a continuous random variable. James V. Lambers Statistical Data Analysis 55 / 56

56 Continuous Exponential Distribution Normal Distribution Example Approximation of the binomial distribution with n = 30 and p = 0.25 (blue circles) by N (np, np(1 p)) (red curve) James V. Lambers Statistical Data Analysis 56 / 56