Determining the Spread of a Distribution

Transcription

1 Determining the Spread of a Distribution Cathy Poliak, Ph.D. cathy@math.uh.edu Department of Mathematics University of Houston Lecture Lecture / 58

2 Outline 1 Describing Quantitative Variables 2 Measurements of Spread 3 Percentiles 4 Quartiles 5 The 1.5IQR Rule 6 Understanding Standard Deviation 7 Calculating The Standard Deviation 8 Coefficient of Variation Lecture / 58

3 Describing distributions of quantitative variables The distribution of a variable tells us what values it takes and how often it takes these values. There are four main characteristics to describe a distribution: 1. Shape 2. Center 3. Spread 4. Outliers Lecture / 58

4 Describing distributions An initial view of the distribution and the characteristics can be shown through the graphs. Then we use numerical descriptions to get a better understanding of the distributions characteristics. Lecture / 58

5 Parameters and Statistics A parameter is a number that describes the population. A parameter is a fixed number, but in practice we usually do not know its value. A statistic is a number that describes a sample. The value of a statistic is known when we have taken a sample, but it can change from sample to sample. We often use a statistic to estimate an unknown parameter. The purpose of sampling or experimentation is usually to use statistics to make statements about unknown parameters, this is called statistical inference. Lecture / 58

6 Notation of Parameters and Statistics Name Statistic Parameter mean x µ mu standard deviation s σ sigma correlation r ρ rho regression coefficient b β beta proportion ˆp p Lecture / 58

7 Describing Center with Numbers Recap Mean: Sum all of the numbers then divide by number of values in the data set. Median: The center value of ordered data. Mode: The value that has the highest frequency in the data. Lecture / 58

8 Example The following are ages of automobiles. Find the mean, median and mode of the age Lecture / 58

9

10 Example: Test Scores The test scores of a class of 20 students have a mean of 71.6 and the test scores of another class of 14 students have a mean of Find the mean of the combined group. Cathy Poliak, Ph.D. cathy@math.uh.edu (Department of Mathematics 1.2 & 1.5 University of Houston ) Lecture 2 33 / 36

11 Example: Conclusions A businesswoman calculates that the median cost of the five business trips that she took in a month is $600 and concludes that the total cost must have been $3000. Explain why the conclusion drawn is not valid. Cathy Poliak, Ph.D. cathy@math.uh.edu (Department of Mathematics 1.2 & 1.5 University of Houston ) Lecture 2 34 / 36

12 Average Test Scores? What is the mean and median for each of these sections test scores? Section A Section B Lecture / 58

13 Types of Measurements for the Spread Range Percentiles Quartiles IQR; Interquartile range Variance Standard deviation Coefficient of Variation Lecture / 58

14 The Range The range is the difference between the highest and lowest values. Section A: Range = = 12 Section B: Range = = 58 Lecture / 58

15 Percentiles The pth percentile of data is the value such that p percent of the observations fall at or below it. The use of percentiles to report spread when the median is our measure of center. If you are looking for the measurement that has a desired percentile rank, the 100P th percentile, is the measurement with rank (or position in the list) of np + 0.5, where n represents the number of data values in the sample. Lecture / 58

16 The 90th percentile of Section A test scores 1. Arrange the scores in order from lowest to highest n = 10, P = 0.90, so the 90 th percentile for this list is at np = 10(0.9) = 9.5, the mean of the 9th and 10th place values. 3. The 90th percentile is = 77 Find the 35th percentile. Find the 75th percentile. Lecture / 58

17 Determine the 25th percentile of the Course Scores Another way to determine percentiles is using the cumulative frequency polygon to estimate percentiles. Cumulative Frequency Chart Cumulative Proportion Scores Lecture / 58

18 Determining Percentiles Suppose you know the position (order) of a value and want to know what percentile it is ranked at. If you have n data measurements, x i represents the 100(i 0.5)/n th percentile. Example: Determine the percentile of the 4 th order statistic for a sample size of n = 15. Lecture / 58

19 Examples of percentiles Suppose you want to know what percentile you are in a certian class. You know there are 200 students in this class and that 20 of the students have scores above you. What is your percentile? Suppose your percentile came out to be 90th percentile, how many students scored the same as or below you? What about at the 50th percentile? Lecture / 58

20 The Quartiles The first quartile is 25th percentile, Q 1. The second quartile is the median and the 50th percentile, Q 2. The third quartile is the 75th percentile, Q 3. Lecture / 58

21 Determining Q 1 for Basketball Shoe Prices Arrange in order n = Q 1 : P = 0.25 np = 15(0.25) = Since we do not get an integer, we find the mean of the 4th and 5th element in the ordered dataset. Q 1 = = 130. Lecture / 58

22

23 Determine Q 2 for Basketball Shoe Prices Arrange in order n = Q 2 : P = 0.5 np = 15(0.5) = 8. So Q 2 is the 8th element of the ordered data. Q 2 = 150. Lecture / 58

24 Determine Q 3 for Basketball Shoe Prices Arrange in order n = Q 3 : P = 0.75 np = 15(0.75) = Again since we did not get and integer, the third quartile is the mean of the 11th and 12th elements in the ordred data. Q 3 = = 215. Lecture / 58

25 R-code for finding Q 1, Q 2, & Q 3 The values: Minimum, Q 1, Median (Q 2 ), Q 3, and Maximum are called the Five Number Summary > shoeprice=c(100,110,120,120,140,140,140,150, 185,185,215,215,250,250,290) > fivenum(shoeprice) [1] Lecture / 58

26 Interquartile Range Interquartile range, IQR, is the difference between Q 3 and Q 1 IQR = Q 3 Q 1 Lecture / 58

27 Example Twelve babies spoke for the first time at the following ages (in months): Find Q 1, Q 2, Q 3, the range and the IQR. Lecture / 58

28 Find the Five Number Summary of the Course Scores > stem(grades$score,scale=0.5) The decimal point is 1 digit(s) to the right of the Lecture / 58

29 Detecting Outliers: 1.5IQR Rule An outlier is an observation that is "distant" from the rest of the data. Outliers can occur by chance or by measurement errors. Any point that falls outside the interval calculated by Q 1 1.5(IQR) and Q (IQR) is considered an outlier. Lecture / 58

30 Outliers for Basketball Shoe Prices? Recall: Q 1 = 130, Q 3 = 215, So IQR = = 85. Q 1 1.5(IQR) = (85) = 2.5 Q (IQR) = (85) = Any price that is below $2.50 or above $ is considered an outlier. Lecture / 58

31 Outliers? The following is information from 91 pairs of basketball shoes: > fivenum(shoes$price) [1] The highest four numbers in the dataset is..., 170, 225, 250, 250. Are there any prices that are considered an outlier? Lecture / 58

32 A Graph of the Five Number Summary: Boxplot A central box spans the quartiles. A line inside the box marks the median. Lines extend from the box out to the smallest and largest observations. Asterisks represents any values that are considered to be outliers. Boxplots are most useful for side-by-side comparison of several distributions. Rcode: boxplot(dataset name$variable name) Lecture / 58

33 Boxplot of Prices boxplot(shoes$price,horizontal = T) Lecture / 58

34 Boxplot of Course Scores Lecture / 58

35 Boxplot of Course Scores by Session Fal15 Sp16 Sum boxplot(grades$score~grades$session,horizontal=true) Lecture / 58

36 Question about the Graphs Given the first type of plot indicated in each pair, which of the second plots could not always be generated from it? a) dot plot, histogram b) stem and leaf, dot plot c) histogram, stem and leaf d) dot plot, box plot Lecture / 58

37 Measuring Spread: The Standard Deviation Measures spread by looking at how far the observations are from their mean. Most common numerical description for the spread of a distribution. A larger standard deviation implies that the values have a wider spread from the mean. Denoted s when used with a sample. This is the one we calculate from a list of values. Denoted σ when used with a population. This is the "idealized" standard deviation. The standard deviation has the same units of measurements as the original observations. Lecture / 58

38 Definition of the Standard Deviation The standard deviation is the average distance each observation is from the mean. Using this list of values from a sample: 3, 3, 9, 15, 15 The mean is 9. By definition, the average distance each of these values are from the mean is 6. So the standard deviation is 6. Lecture / 58

39 Definition of the Standard Deviation The standard deviation is the average distance each observation is from the mean. Using this list of values from a sample: 3, 3, 9, 15, 15 The mean is 9. By definition, the average distance each of these values are from the mean is 6. So the standard deviation is 6. Lecture / 58

40 Definition of the Standard Deviation The standard deviation is the average distance each observation is from the mean. Using this list of values from a sample: 3, 3, 9, 15, 15 The mean is 9. By definition, the average distance each of these values are from the mean is 6. So the standard deviation is 6. Lecture / 58

41 Definition of the Standard Deviation The standard deviation is the average distance each observation is from the mean. Using this list of values from a sample: 3, 3, 9, 15, 15 The mean is 9. By definition, the average distance each of these values are from the mean is 6. So the standard deviation is 6. Lecture / 58

42 Values of the Standard Deviation The standard deviation is a value that is greater than or equal to zero. It is equal to zero only when all of the observations have the same value. By the definition of standard deviation determine s for the following list of values. 2, 2, 2, 2 : standard deviation = 0 125, 125, 125, 125, 125: standard deviation = 0 Lecture / 58

43 Values of the Standard Deviation The standard deviation is a value that is greater than or equal to zero. It is equal to zero only when all of the observations have the same value. By the definition of standard deviation determine s for the following list of values. 2, 2, 2, 2 : standard deviation = 0 125, 125, 125, 125, 125: standard deviation = 0 Lecture / 58

44 Values of the Standard Deviation The standard deviation is a value that is greater than or equal to zero. It is equal to zero only when all of the observations have the same value. By the definition of standard deviation determine s for the following list of values. 2, 2, 2, 2 : standard deviation = 0 125, 125, 125, 125, 125: standard deviation = 0 Lecture / 58

45 Values of the Standard Deviation The standard deviation is a value that is greater than or equal to zero. It is equal to zero only when all of the observations have the same value. By the definition of standard deviation determine s for the following list of values. 2, 2, 2, 2 : standard deviation = 0 125, 125, 125, 125, 125: standard deviation = 0 Lecture / 58

46 Values of the Standard Deviation The standard deviation is a value that is greater than or equal to zero. It is equal to zero only when all of the observations have the same value. By the definition of standard deviation determine s for the following list of values. 2, 2, 2, 2 : standard deviation = 0 125, 125, 125, 125, 125: standard deviation = 0 Lecture / 58

47 Values of the Standard Deviation The standard deviation is a value that is greater than or equal to zero. It is equal to zero only when all of the observations have the same value. By the definition of standard deviation determine s for the following list of values. 2, 2, 2, 2 : standard deviation = 0 125, 125, 125, 125, 125: standard deviation = 0 Lecture / 58

48 Values of the Standard Deviation The standard deviation is a value that is greater than or equal to zero. It is equal to zero only when all of the observations have the same value. By the definition of standard deviation determine s for the following list of values. 2, 2, 2, 2 : standard deviation = 0 125, 125, 125, 125, 125: standard deviation = 0 Lecture / 58

49 Adding or Subtracting a Value to the Observations Adding or subtracting the same value to all the original observations does not change the standard deviation of the list. Using this list of values: 3, 3, 9, 15, 15 mean = 9, standard deviation = 6. If we add 4 to all the values: 7, 7, 13, 19, 19 mean = 13, standard deviation = 6 Lecture / 58

50 Adding or Subtracting a Value to the Observations Adding or subtracting the same value to all the original observations does not change the standard deviation of the list. Using this list of values: 3, 3, 9, 15, 15 mean = 9, standard deviation = 6. If we add 4 to all the values: 7, 7, 13, 19, 19 mean = 13, standard deviation = 6 Lecture / 58

51 Adding or Subtracting a Value to the Observations Adding or subtracting the same value to all the original observations does not change the standard deviation of the list. Using this list of values: 3, 3, 9, 15, 15 mean = 9, standard deviation = 6. If we add 4 to all the values: 7, 7, 13, 19, 19 mean = 13, standard deviation = 6 Lecture / 58

52 Adding or Subtracting a Value to the Observations Adding or subtracting the same value to all the original observations does not change the standard deviation of the list. Using this list of values: 3, 3, 9, 15, 15 mean = 9, standard deviation = 6. If we add 4 to all the values: 7, 7, 13, 19, 19 mean = 13, standard deviation = 6 Lecture / 58

53 Multiplying or Dividing a Value to the Observations Multiplying or dividing the same value to all the original observations will change the standard deviation by that factor. Using this list of values: 3, 3, 9, 15, 15: mean = 9, standard deviation = 6. If we double all the values: 6, 6, 18, 30, 30 mean = 18, standard deviation = 12 Lecture / 58

54 Multiplying or Dividing a Value to the Observations Multiplying or dividing the same value to all the original observations will change the standard deviation by that factor. Using this list of values: 3, 3, 9, 15, 15: mean = 9, standard deviation = 6. If we double all the values: 6, 6, 18, 30, 30 mean = 18, standard deviation = 12 Lecture / 58

55 Multiplying or Dividing a Value to the Observations Multiplying or dividing the same value to all the original observations will change the standard deviation by that factor. Using this list of values: 3, 3, 9, 15, 15: mean = 9, standard deviation = 6. If we double all the values: 6, 6, 18, 30, 30 mean = 18, standard deviation = 12 Lecture / 58

56 Multiplying or Dividing a Value to the Observations Multiplying or dividing the same value to all the original observations will change the standard deviation by that factor. Using this list of values: 3, 3, 9, 15, 15: mean = 9, standard deviation = 6. If we double all the values: 6, 6, 18, 30, 30 mean = 18, standard deviation = 12 athy Poliak, Ph.D. cathy@math.uh.edu (Department of Mathematics University of Houston ) Lecture / 58

57 Population Variance and Standard Deviation If N is the number of values in a population with mean mu, and x i represents each individual in the population, the the population variance is found by: σ 2 = N i=1 (x i µ) 2 N and the population standard deviation is the square root, σ = σ 2. Lecture / 58

58 Sample Variance and Standard Deviation Most of the time we are working with a sample instead of a population. So the sample variance is found by: s 2 = n i=1 (x i x) 2 n 1 and the sample standard deviation is the square root, s = s 2. Where n is the number of observations (samples), x i is the value for the i th observation and x is the sample mean. Lecture / 58

59 Calculating the Standard Deviation By Hand When calculating by hand we will calculate s. 1. Find the mean of the observations x. 2. Calculate the difference between the observations and the mean for each observation x i x. This is called the deviations of the observations. 3. Square the deviations for each observation (x i x) Add up the squared deviations together n i=1 (x i x) Divide the sum of the squared deviations by one less than the number of observations n 1. This is the variance s 2 = 1 n 1 n (x i x) 2 i=1 Lecture / 58

60 Calculating the Standard Deviation By Hand When calculating by hand we will calculate s. 1. Find the mean of the observations x. 2. Calculate the difference between the observations and the mean for each observation x i x. This is called the deviations of the observations. 3. Square the deviations for each observation (x i x) Add up the squared deviations together n i=1 (x i x) Divide the sum of the squared deviations by one less than the number of observations n 1. This is the variance s 2 = 1 n 1 n (x i x) 2 i=1 athy Poliak, Ph.D. cathy@math.uh.edu (Department of Mathematics University of Houston ) Lecture / 58

61 Calculating the Standard Deviation By Hand When calculating by hand we will calculate s. 1. Find the mean of the observations x. 2. Calculate the difference between the observations and the mean for each observation x i x. This is called the deviations of the observations. 3. Square the deviations for each observation (x i x) Add up the squared deviations together n i=1 (x i x) Divide the sum of the squared deviations by one less than the number of observations n 1. This is the variance s 2 = 1 n 1 n (x i x) 2 i=1 athy Poliak, Ph.D. cathy@math.uh.edu (Department of Mathematics University of Houston ) Lecture / 58

62 Calculating the Standard Deviation By Hand When calculating by hand we will calculate s. 1. Find the mean of the observations x. 2. Calculate the difference between the observations and the mean for each observation x i x. This is called the deviations of the observations. 3. Square the deviations for each observation (x i x) Add up the squared deviations together n i=1 (x i x) Divide the sum of the squared deviations by one less than the number of observations n 1. This is the variance s 2 = 1 n 1 n (x i x) 2 i=1 athy Poliak, Ph.D. cathy@math.uh.edu (Department of Mathematics University of Houston ) Lecture / 58

63 Calculating the Standard Deviation By Hand When calculating by hand we will calculate s. 1. Find the mean of the observations x. 2. Calculate the difference between the observations and the mean for each observation x i x. This is called the deviations of the observations. 3. Square the deviations for each observation (x i x) Add up the squared deviations together n i=1 (x i x) Divide the sum of the squared deviations by one less than the number of observations n 1. This is the variance s 2 = 1 n 1 n (x i x) 2 i=1 athy Poliak, Ph.D. cathy@math.uh.edu (Department of Mathematics University of Houston ) Lecture / 58

64 Step 6: Standard Deviation 6. Find the square root of the variance. This is the standard deviation s = 1 n (x i x) n 1 2 i=1 Lecture / 58

65 Example: Section A Determine the sample standard deviation of the test scores for Section A. Section A Scores (X i ) Lecture / 58

66 Step 1: Calculate the Mean The sample mean is x = Lecture / 58

67 Use Table To Calculate Standard Deviation Variable Deviations Deviations Squared Score (X i ) X i X (X i X) sum Lecture / 58

68 Step 2: Calculate Deviations For All Values Variable Deviations Deviations Squared Score (X i ) X i X (X i X) = = = = = = = = = = 5.5 sum Lecture / 58

69 Step 3: Calculate Squared Deviations Variable Deviations Deviations Squared Score (X i ) X i X (X i X) = 6.5 ( 6.5) 2 = = 5.5 ( 5.5) 2 = = 4.5 ( 4.5) 2 = = 3.5 ( 3.5) 2 = = 0.5 ( 0.5) 2 = = = = = = = = = = = sum Lecture / 58

70 Step 4: Calculate the Sum of the Squared Deviations Variable Deviations Deviations Squared Score(X i ) X i X (X i X) = 6.5 ( 6.5) 2 = = 5.5 ( 5.5) 2 = = 4.5 ( 4.5) 2 = = 3.5 ( 3.5) 2 = = 0.5 ( 0.5) 2 = = = = = = = = = = = sum n i=1 (X i X) 2 = Lecture / 58

71 Step 5: Calculate the Variance variance = s 2 = 1 n 1 n (x i x) 2 i=1 = = Lecture / 58

72 Step 6: Take the Square Root of the Variance standard deviation = s = 1 n 1 = = 4.77 n (x i x) 2 i=1 Lecture / 58

73 Sample Standard Deviation of Section A test scores Sample standard deviation is s = This implies that from the sample of the 10 students from section A the tests scores has a spread, on average, of 4.77 points from the mean of points. Lecture / 58

74 Example A statistics teacher wants to decide whether or not to curve an exam. From her class of 300 students, she chose a sample of 10 students and their grade were: 72, 88, 85, 81, 60, 54, 70, 72, 63, 43 Determine the sample mean. What is the variance? What is the standard deviation? Lecture / 58

75 Add 10 Suppose the statistics instructor decides to curve the grade by adding 10 points to each score. What is the new mean, variance and standard deviation? Lecture / 58

76 Multiply by 2 For the following dataset the mean is x = 4.5, the variance is s 2 = 3.5 and the standard deviation is s = , 6, 2, 7, 4, 5 Now, multiply each value by 2. What is the new variance and the new standard deviation? Lecture / 58

77 Calculating Standard Deviation For larger data sets use a calculator or computer software. Each calculator is different if you cannot determine how to compute standard deviation from your calculator ask your instructor. For this course we will be using R as the software. The function for the sample standard deviation in R is sd(data name$variable name). Lecture / 58

78 Coefficient of Variation This is to compare the variation between two groups. The coefficient of variation (cv) is the ratio of the standard deviation to the mean. cv = sd mean A smaller ratio will indicate less variation in the data. Lecture / 58

79 CV of test scores Section A Section B Sample Size Sample Mean Sample Standard Deviation CV 71.5 = = Lecture / 58

80 CV Example The following statistics were collected on two different groups of stock prices: Portfolio A Portfolio B Sample size Sample mean $52.65 $49.80 Sample standard deviation $6.50 $2.95 What can be said about the variability of each portfolio? Lecture / 58

81 Things to do before Thursday 1. Try to download R and R-studio 2. Start working on homework 1 3. Work on quiz 1. Lecture / 58