Last time. Numerical summaries for continuous variables. Center: mean and median. Spread: Standard deviation and inter-quartile range

Size: px

Start display at page:

Download "Last time. Numerical summaries for continuous variables. Center: mean and median. Spread: Standard deviation and inter-quartile range"

Cory Conley
5 years ago
Views:

1 Lecture 4

2 Last time Numerical summaries for continuous variables Center: mean and median Spread: Standard deviation and inter-quartile range Exploratory graphics Histogram (revisit modes )

3 Histograms Histogram of income Frequency e00 1e05 2e05 3e05 4e05 income

4 Histograms: Skew (heavy right tail)

5 Histograms: Skew bimodal

6 Histograms: Trimodal

7 Histograms: Normal (Bell curve)

8 Histograms: Symmetric

9 Histograms: Symmetric (with four modes)

10 5-number summary (Min, Lower Q, Med, Upper Q, Max) Example: Average daily temperatures in Philadelphia ( ), n=5479 Min Lower Qu Median Upper Qu Max

11 Temp Boxplots Built around the 5-number summary Box constructed from lower and upper quartiles Median marked with a horizontal bar Whiskers designed to capture most of the data (99.5% for normal or bell shape) Points for data beyond whiskers, possible outliers

12 Mortality Boxplots Whiskers Find the largest value that is within 1.5 IQR of the upper quartile; mark that point with a bar Find the smallest value that is no smaller than 1.5 IQR below the lower quartile Points for data beyond whiskers, possible outliers Mortality summary: Min Lower Qu Median Upper Qu Max

13 Histograms of Daily Mortality Histogram of non accidental mortality Frequency Number of deaths

14 More boxplots: Non-accidental mortality Daily mortality fall winter spring summer Season

15 Histograms of Daily Temperatures Histogram of average daily temperatures Frequency Temperatures

16 Boxplots of Daily Temperatures Average daily temperature fall winter spring summer Season

17 A similar idea: Conditioning summer winter Count fall spring Temperature

18 Summaries for discrete or qualitative data Frequency tables and bar graphs Example: BRFSS What is the highest grade or year of school you completed? freq % cumu % None Gd Gd Gd Col Col Refused n=108,

19 Barplots (note difference from histograms) None Gd 1 8 Gd 9 11 Gd 12 Col1 3 Col 4 Refused

20 Barplots None Gd 1 8 Gd 12 Col1 3 Col 4 Refused

21 Two continuous variables Scatter plots can display the relationship between two continuous variables (conditioning can give more control) Can be used to spot trends in the data, relationships as well as their strength Random versus controlled variables in the plot

22 Scatter plot (BRFSS) Weight Height

23 Scatter plot (BRFSS) Height Weight

24 Scatter plot (Time series of mortality) Daily mortality Date

25 Scatter plot (Time series of temperature) Daily temperature Date

26 Lecture 5

27 Last time Graphics for exploratory analysis Histograms, boxplots, barplots, scatterplots When faced with several variables... Interactive analysis Conditioning when creating plots How s the lab?

28 Probability Toss a coin. What is the probability of heads? Roll a die. What is the probability of getting a five?

29 Some historical examples Count Buffon ( ) tossed a coin 4,040 times, with heads coming up 2,048 or percent of the time Karl Pearson ( ) tossed a coin 24,000 times with heads coming up 12,012 or 50.05% of the time John Kerrich tossed a coin 10,000 times (while he was a prisoner of war in WWII) and heads came up 5,067 times or percent

30 Some historical examples F.N. David and Roman dice, 204 tosses Rock crystal 30, 38, 31, 34, 34, 37 Iron 35, 39, 30, 21, 37, 42 Marble 27, 28, 23, 47, 25, 54

31 Some historical examples Diaconis mechanical coin flipper Coin always lands the same way

32 Computer experiment relative frequency flip number

33 Probability and relative frequency Toss a coin many, many times Examine the proportion of flips that turn up heads What should you get?

34 Probability and relative frequency Perform a large number of independent repetitions of a random phenomenon After each trial, record the proportion of times in which an event occurs This relative frequency approaches a fixed number that we call the probability of the event

35 Probability models With coins or dice, we often speak of fairness, or rather that there is no preference for one event over another This is an idealization of a random phenomenon Probability models are mathematical descriptions that account for unpredictable factors in random events

36 Three kinds of probabilities Probabilities from models Probabilities from data (relative frequencies) Subjective probabilities (beliefs)

37 Some terminology A random experiment is some situation with an unpredictable outcome The sample space of an experiment is the set of all possible outcomes An event is a collection of outcomes

38 Some terminology Random experiments: Tossing a coin, measuring your blood pressure, taking a test Sample space: H/T, (whatever BP is measured in), A-F grade Event: The coin turns up heads, my blood pressure is in the normal range, I pass the exam

39 Some notation Sample space is denoted by the symbol S Tossing a coin 2 times, S = {HH,HT,TH,TT} Events (your text uses labels and names events or simply assigns symbols A, B, C...) Tossing at least one head, A={HH,HT,TH}

40 Lecture 6

41 Last time Conceptual definition of probability Long-run averages of outcomes from repeated independent experiments Probability models Mathematical descriptions that account account for unpredictable factors in random events Terminology and notation describing events

42 Working with events The complement of an event A occurs if A does not occur; we denote it by A We combine events with set operations of intersection and union, and and or Two events are mutually exclusive if they cannot occur at the same time A B

43 Sample space and events S A = {even spots} A

44 Sample space and events S B = {fewer than 5 spots} A

45 Complement B = {fewer than 5 spots} B A

46 Intersection (and) B = {fewer than 5 spots} A = {even spots} A

47 Intersection (and) A and B = {fewer than 5 spots and even} A

48 Union (or) B = {fewer than 5 spots} A = {even spots} A

49 Sample space and events A or B = {Fewer than 5 spots or even} A

50 Probability distributions For a sample space assign values S = S 1, S 2, S 3,... p 1, p 2, p 3,... we Each value is between 0 and 1 The sum of the values adds to one 1 = p 1 p 2 p 3... The probability of an event A is the sum of the values for all the outcomes in A

51 Equally likely outcomes By symmetry we can might believe no one outcome occurs more frequently than any other The probability of an event A is then pr(a) = Number of outcomes in A Total number of outcomes in S

52 S B = {fewer than 5 spots} pr(b) = 4/6 = 2/3 A

53 Rules for working with probabilities The sample space is certain to occur pr(s) = 1 pr(a does not occur) = 1-pr(A does occur) pr(a) = 1-pr(A) The probability of two mutually exclusive events occurring is the sum of the two events pr(a or B) = pr(a) pr(b)

54 Mutually exclusive events B = {throw a 1} A = {throw a 6} A pr(a or B) = 1/6 1/6 = 1/3

55 Rules for working with probabilities Addition rule holds for any number of mutually exclusive events pr(a 1 or A 2 or... or A k ) = pr(a 1 ) pr(a 2 ) pr(a k )

56 ... and if they re not mutually exclusive? When two events are not mutually exclusive, we have double counted the intersection; instead we use the rule pr(a or B) = pr(a) pr(b) - pr(a and B)

57 Screening tests These are tests designed to determine if someone might have a particular medical condition; typically they are applied to a large segment of the population and the positives are subjected to further diagnostic tests

58 Screening tests and a 2-by-2 table Disease status Y N Pos sick and pos well and pos Test result Neg sick and neg well and neg

59 Screening tests and a 2-by-2 table Disease status Y N Pos pr(sick and pos) pr(well and pos) pr(pos) Test result Neg pr(sick and neg) pr(well and neg) pr(neg) pr(sick) pr(well)

60 Conditional probability The conditional probability of A occurring given that B has occurred is denoted pr(a B) and is defined by pr(a and B) / pr(b) This gives us the multiplicative rule for probabilities p(a and B) = P(A B) pr(b)

61 Conditional probability Disease status Y N Pos pr(sick and pos) pr(well and pos) pr(pos) Test result Neg pr(sick and neg) pr(well and neg) pr(neg) pr(sick) pr(well)

62 Conditional probability The conditional probability that a test is positive given that someone is sick is pr(positive and sick) / pr(sick) What is the probability that someone is sick given that the test is negative?

CHAPTER 5: EXPLORING DATA DISTRIBUTIONS. Individuals are the objects described by a set of data. These individuals may be people, animals or things.

(c) Epstein 2013 Chapter 5: Exploring Data Distributions Page 1 CHAPTER 5: EXPLORING DATA DISTRIBUTIONS 5.1 Creating Histograms Individuals are the objects described by a set of data. These individuals