Data Analysis and Statistical Methods Statistics 651

Transcription

1 Data Analysis and Statistical Methods Statistics (Chapter 2) Lecture 5 (MWF) Probabilities and the rules Suhasini Subba Rao

2 Review of previous lecture We looked at the interquartile range. The concept of a variance and standard deviation was introduced. The standard deviation is the square root of the variance. Like the mean, the standard deviation (which is the square root of the variance) is a measure of spread of the population, and is not random. On the other hand, the sample variance and standard deviation are random. In statistics we have: Populations, Random variables (the measurements we are interested in the population) and Probabilities which are allocated 1

3 to the random variables. In this lecture we will introduce the notion of a probability and the rules that go with it. 2

4 Probability Suppose there is a population of people and I select one person at random. What is the chance (probability) that the selected person s height lies in the interval [5.5, 5.75]feet? How do we calculate this chance? The proportion of heights that lie in [5.5, 5.75] is: Total number of people in population with height in the interval [5.5, 5.75]. Total number of people in population 3

5 Summary: Probabilities and random variables A probability is always positive and is greater than or equal to zero and less than or equal to one (lies in the interval [0, 1]). Random variables and probabilities A random variable (often denoted X) is a variable whose outcomes are random (eg. the height of a randomly selected student or the gender of a random selected person) A random variable can take any one of several different outcomes (this is why it is random), to each outcome we allocate a probability. We now consider some examples to familiarize ourselves with this notation. 4

6 Examples of random variables and their probabilities Y is the (assume for simplicity binary) gender of a randomly chosen in the world population. Then Y = {Male or Female}. We write P (Y =Female) to denote the probability a random chosen person is female (equivalently the proportion of females). X is the height of a randomly chosen person. P (5 < X < 5.25), denotes the probability of that a randomly selected students height is in the interval [5, 5.25] (equivalently the proportion of people between feet). 5

7 Illustration:Data from a 651 class Summary of information from a 651 class. 6

8 Probabilities and random variables: Illustration 1 (i) The measurement of interest may be the height of the randomly chosen person. Therefore X = height of randomly selected person. The set of all possible outcomes is any number in the interval [5, 7]. Therefore, P (5 X 7) = 1. P (5.75 X < 6.25) = P (5.75 X < 6) + P (6 X < 6.25) = It is impossible from this plot to evaluate P (5.9 X < 6.15). (ii) The measurement of interest may be the gender of the randomly chosen person. Therefore Y = gender of randomly chosen person. The set of all possible outcomes is {male, female}. We see that P (Y = Female) = and P (Y = Male) =

9 We recall that data set: Lecture 5 (MWF) Random variables, Probability and conditional probabilities Illustration 2: Binge drinking Male Female Binge Not Binge Subtotals P (Binge) = 3314/17096 = P (Not Binge) = 13782/17076 =

10 Mutually exclusive events Two events (outcomes) are mutually exclusive, if the occurrence of one event excludes the occurrence of the other event. In the height example, we define the events A = [5, 5.5) is an (this is the event that a randomly chosen person s height lies in the interval A = [5, 5.5)) and B = [5.75, 6). A = [5, 5.5) and B = [5.75, 6) are mutually exclusive events. This means if person s height is in A = [5, 5.5), then their height cannot be in B = [5.75, 6]. Similarly, if X is in B = [5.75, 6) then it cannot be in A = [5, 5.5). On the other hand, the events A = [5, 5.5) and C = [5.25, 5.75) are not mutually exclusive. If X = 5.3, then it is in both A and C. 9

11 Let X denote the time of the day a person wakes up. Let A=night time and B=day time, then A and B are mutually exclusive events. In any given day one the person cannot get up at both night and day time. 10

12 Mutually exclusive events and probabilities Calculating probabilities If two events A and B are mutually exclusive, then P (A or B) = P (A) + P (B) (the probability of either A or B arising). Binge Drinking example In the survey above students were asked if they had every drunk in excess. The answer is binary, yes or no. Yes and No are mutually exclusive events. Since P (Yes or No) = P (Yes) + P (No) = 1. We saw from above that P (Binge)= P (Yes) = 3314/17096 = This means P (No) = =

13 Height Example Suppose X is the height of a randomly chosen person. If A = [5, 5.5) and B = [5.75, 6), then P (X in A or X in B) = P (5 X < 5.5 or 5.75 X < 6). Since A and B are mutually exclusive then P (X in A or X in B) = P (5 X < 5.5 or 5.75 X < 6) = P (5 X < 5.5) + P (5.75 X < 6). Plot events A and B on a density plot. 12

14 Example 2 Let X denote the height of a person. Suppose that the probability P (X t) is known for all t. We know that students heights lie within 5 7 feet. Using this information, how can we calculate the probabilities below using P (X t) for any t: (i) P (5.5 X < 6.5). (ii) P (X > 6). (iii) P (5 X < 6 or 6.5 X < 7). (iv) P (5 X 6 or 5.75 X < 7). (v) P (5 X < 6 and 5.75 X < 7). We use this formulation to calculate probabilites using the normal/z tables. 13

15 Solution 2 (i) P (5.5 < X 6.5). 14

16 (ii) P (X > 6) = 1 P (X 6). Lecture 5 (MWF) Random variables, Probability and conditional probabilities 15

17 (iii) P (5 < X 6 or 6.5 < X 7). Lecture 5 (MWF) Random variables, Probability and conditional probabilities 16

18 (iv) P (5 < X 6 or 5.5 < X 7). Lecture 5 (MWF) Random variables, Probability and conditional probabilities 17

19 (v) P (5 < X 6 and 5.5 < X 7). Lecture 5 (MWF) Random variables, Probability and conditional probabilities 18

20 Motivation: Conditional probabilities Is there a relationship between binge drinking (excessive consumption of alcohol) and gender students were surveyed (7180 males and 9916 females). The data is given below. Male Female Binge Not Binge Subtotals In order to analyze this type of data we need to introduce the notion of a conditional probability. This is a probability calculated using not the entire population but a subpopulation. 19

21 Motivation: cont A conditional probability is the probability the probability of an event given that we already have some (possibly partial) information about. Male Female Binge Not Binge Subtotals Proportion of females who were bing drinking is 1684/ %, whereas proportion of males who were binge drinking is 1630/ %. We can write these conditional probabilities as P(binge drink Male) = and P(binge drink Female) = Based on the above information, does additional information about gender change the chance of binge drinking? 20

22 This example alludes to the notion of independence, that we define at the end of this lecture. 21

23 Stents In a randomized trial 451 at risk stroke patients were randomly placed in a stents (medical devise) group and a control group. The data after one year is summarized below. Stroke No event treatment control Subtotals P (stroke stent) = 45/224 = 0.2 P (stroke control) = 28/227 = 0.12 What does the difference suggest. Why should we be careful about drawing conclusions about the population of at risk stroke patients based on just the difference of 0.2 and 0.12? 22

24 Conditional probability: Heights and gender of 18 people Height Gender F M F M M F F M F Height Gender M M F M F M M F F We randomly select a person. Let X denote their height and Y their gender. Calculate (i) P (X 5.5). (ii) P (X 5.5 Y = M). (iii) P (X 5.5 Y = F ). What do we observe? Is there a difference in the distribution of male and female heights. 23

25 Conditional probability Using contingency tables to calculate conditional probabilities is straighforward. However, sometimes the variable on which we condition is not always categorical. In this case we use the formula: P (A B) = P (A and B). P (B) The same formula can also be used for contingency tables. Keep in mind that P (A and B) means that an observation has to be in both sets A and B. For example P (5 X < 6 and 5.75 X < 7) = P (5.75 X < 6). 24

26 Whereas P (A or B) means that an observation can satisfy either one of the conditions. Example 1 (height) Suppose X is the height of a randomly selected person and we know that X lies in the interval [5, 6], then what is the probability X lies in [5, 5.5]? We write this probability as: Number of people whose height is in [5, 5.5] Number of people whose height is in [5, 6] = P (5 X X 6) Answer: Using the formula we calculate the probability. Let A = 5 X 5.5 and B = 5 X 6. Using 25

27 We have P (A B) = P (A and B) P (B) = P (A) P (B) 26

28 P (5 X X 6) = = = We see that P (5 X X 6) > P (5 X 5.5). In this example, additional information about the height increased the chance. 27

29 Additional information will not always increase the chance. For example, P (X > 6 5 X 6) = 0 < P (5 X 5.5). Example 2 (height) Calculate the the probability X is between 5 and 6 feet given that we know that person s height is between feet (P (5 < X 6 5 < X 5.5))? Answer Using the formula we calculate the probability. Let A = 5 X 6 and B = 5 X 5.5. Using We have P (5 < X 6 5 < X 5.5) = P (A B) = P (B) P (B) = 1. 28

30 Independence and conditional probabilities Definition Suppose that we have two events A and B. The events A and B are independent of each other if P (A B) = P (A). This means the event B has no influence what so ever on the chance of event A occurring. Let us return to the example of heights and gender. Are height and gender independent variables? In other words, does information about the gender of a person give additional information about their height. Intuitively this seems to be the case. We showed in a previous example: P (5 X 5.5 Y = female) > P (5 X 5.5 Y = male). Which means that P (5 X 5.5 Y = female) P (5 X 5.5) This implies that gender and height are not independent variables. 29

31 Random variables X and Y are independent if for any outcome of X and and outcome of Y, P (X = event A Y = event B) = P (X = event A). Mutually exclusive events are dependent! Example Let A denote the event a height is in the interval [5, 5.5] and B denote the event a height is in the interval [6, 6.5]. We define the corresponding binary variables, Y =Yes if A is true and Y =No if A is not true the variable Z=Yes if B is true and Z=No if B is not true). Then P (Y = True Z=True) = P (X in [5, 5.5] X in [6, 6.5]) = 0 P (X in [5, 5.5]) = P (Y = True). A and B are not independent events. In other words, information about Z gives us additional information about Y. 30

32 Application: Binge drinking Male Female Binge Not Binge Subtotals The proportion of binge drinking in females = P(binge drink Female) = The proportion of binge drinking in males = P(binge drink Male) = By definition there is a dependence/association between between gender and binge drinking (at least in this sample). However, we must be careful. We cannot determine a causality. This is a observational study and other factors (associated with the male participants) such as fraternities, may be driving the increase. 31

33 Application: Stents Stroke No event treatment control Subtotals Proportion of treatment group who got a stroke P (stroke treatment) = 45/224 = 0.2. Proportion in control group who got a stroke P (stroke control) = 28/227 = For this example, there is a dependence between treatment and stroke. If we can determine that this difference is statistically significant (cannot be explained by sample variation), then since the study is experiment with the only difference being the treatment it suggests that the use of stents in increasing the risk of a stroke (a causality can be determined). 32

34 Examples: Independent events Define the random variables. X is the season in college station. Y are the number of hair dressers in college station. Z is the temperature in college station. It is unlikely that the number of hair dressers in college station have an influence on the temperature. Therefore P (Z [25, 28] Y ) = P (X [25, 28]). 33

35 On the other hand season does have an unfluence on temperature, therefore P (X [25, 28] X = summer ) P (X [25, 28]). 34

36 Dependence does not imply causality A study was done in the early 90s to see if the mortality rates between left and right handed people were the same. To do this the psychologists collected the death records of 2000 people who died in May, 1990, in Southern California. They rang the families of all these people and asked whether the were left or right handed. They also categorized the people who died into those above 60 and below 60 years old. This is the data they collected. Out of the 2000, 400 were left handed. Out of the 2000, 150 were left handed and died below 60. Out of the 2000, 300 were right handed and died below

37 (a) A person dies below 60? Lecture 5 (MWF) Random variables, Probability and conditional probabilities (b) A person dies below 60 given that they are left handed? (c) Does the data suggest there is an association/dependence between left handedess and early mortality? 36

38 Solution It instructive to summarize the data as a contingency table: Died before 60 Died after 60 totals Left Right Totals (a) The chance a person dies below 60. Calculate the total proportion who died before 60 P (before 60) = (b) A person dies below 60 given that they are left handed. 37

39 Focus on people who are only left handed. There are 400 of these. Of these, 150 died before 60. Therefore the proportion of left handed people in the sample who died before 60 is P (before 60 left handed) = = 3 8 = 37.5% Using the same argument the proportion of right handed people in the sample who died before 60 is P (before 60 right handed) = = 3 16 = 18.75%. (c) As these proportions are different, it suggests there is some sort of dependence/association between lefthandedness and early mortality. 38

40 Why the dependence? Can we conclude from this data that being left handed increases the chance of early mortality? This statement is a causal statement, it asks whether being left handed causes early mortality. This data set cannot answer this question, does not take into account other variables associated with being left handed which are causing the effect you are seeing. The most plausible explanation for the difference in proportions is that the incidence of left-handedness has increased over the past 50 years (because people are no longer forced to be right handed). The unobserved variable is 39

41 Example: Before 1930, no one was left handed, what would the numbers in the table and proportions be? 40