Exploring Data. How to Explore Data

Transcription

1 Exploring Data Statistics is the art and science of learning from data. This may include: Designing appropriate tools to collect data. Organizing data in a meaningful way. Displaying data with appropriate graphs. Summarizing data with numbers. Using data to make predictions. Data are numbers or other information in context. Individuals are the objects described by a set of data. They may be people, animals, or things. A variable is any characteristic of an individual. It can take different values for different individuals. A categorical (or qualitative) variable places an individual into one of several groups or categories. A quantitative variable takes numerical values for which it makes sense to find an average. Not every variable with a number value is quantitative! Examples: The distribution of a variable shows what values the variable takes and how often it takes each value. How to Explore Data Begin by examining each variable by itself. Then move on to study relationships among the data. Start with a graph or graphs. Then add numerical summaries. Example: The following table shows information about several popular cell phone models. Phone Operating System Screen Size (inches) Internal Storage (GB) Expandable Storage Rear Camera (megapixels) Battery Life (Talk Time) (hours) Apple iphone 6S Plus ios No Apple iphone 6s ios No Apple iphone 6 ios No 8 14 BlackBerry DTEK 50 Android Yes BlackBerry Priv Android Yes BlackBerry Leap BlackBerry Yes 8 25 LG X Skin Android Yes 8 7 LG G5 SE Android Yes LG G5 Android Yes Microsoft Lumia 650 Windows Yes 8 13 Microsoft Lumia 950 Windows Yes Microsoft Lumia 950 XL Windows Yes Samsung Galaxy Note 7 Android Yes Samsung Galaxy On 7 Pro Android Yes Samsung Galaxy S7 Edge Android Yes 12 33

2 a) Who/what are the individuals in this data set? b) What variables are measured? Identify each as categorical or quantitative. In what units were the quantitative variables measured? c) Give the distributions of the following for the data set: screen size, internal storage, and presence of expandable memory. Analyzing Categorical Data The values of a categorical variable are labels for the categories, such as male or female. The distribution of a categorical variable gives the categories and either the count or proportion of individuals who fall into each category. Proportion: The fraction of the total that possesses a certain attribute. Proportions can be expressed as fractions, decimals, or percentages. Frequency: The number (count) of individuals in each category. Relative Frequency: The proportion of individuals in each category. Often, we organize categorical data into either a frequency table or a relative frequency table. Example: The following is a frequency table showing the distribution of responses to the question, How do you eat corn on the cob? Find the relative frequency distribution. How do you eat corn on the cob? Frequency Relative Frequency In rows 22 In circles 13 Bite wherever 6 I don t eat corn on the cob 5 Cut the corn off the cobb 4 Total Categorical data is often displayed using bar graphs and pie charts.

3 Bar Graph Procedure: 1. Draw a horizontal line. Write the category names or labels below the line at regularly spaced intervals. 2. Draw a vertical line. Label the scale using either frequency or relative frequency. 3. Place a rectangular bar above each category label with the height corresponding to the frequency or relative frequency. Make sure all bars have the same width. Bars are historically drawn with gaps between them to indicate that the data is not continuous. Pie Chart Procedure: 1. Draw a circle to represent the entire data set. 2. For each category, calculate the size of the central angle of the slice : slice size= 360 relative frequency of category 3. Use a protractor to divide the circle into slices with the appropriate central angles. 4. Label the slices appropriately! An alternative to a pie chart is a segmented bar graph, where there is a single bar with segments corresponding to the different categories. Segmented bar graphs use relative frequencies on the vertical axis. The height of each segment corresponds to the relative frequency of that category. Example: Draw a well-labeled bar graph, a well-labeled segmented bar graph, and a well-labeled pie chart of the corn data from the previous example. Bar graphs are more flexible than pie charts! Bar graphs can compare any set of quantities that are measured in the same units, but pie charts can only compare all parts of a single whole. For example, you couldn t compare the percentages of sophomores, juniors, and seniors who approve of the school s parking policies with a pie chart. These are not parts of the same whole. The percentages wouldn t add to 100%. You could use a bar graph for this data, however.

4 Deceptive Graphs: Watch out for graphs in which the width changes in addition to the height. The eye responds to area, so this makes the graph misleading. This happens a lot in pictographs. Watch out for graphs where the axes don t start at zero (and/or are missing).

5 Watch out for unequally-spaced intervals. Watch out for pie charts or segmented bar graphs where the percentages don t add to 100%. Pie charts and segmented bar graphs represent all the parts of a single whole. Bar graphs, on the other hand, can represent proportions of different groups, so it is okay if the percentages on a bar graph don t add to 100%. Watch out for 3D graphs or graphs set at an angle. This distorts the data. Perception of 3D Pie Charts 20% 30% 20% 30% Cool Confusing Misleading Unreadable

6 Two-Way Tables (also called Contingency Tables) Often, we are interested in the relationship between two categorical variables. In this case, we can organize the data into a table, with the rows representing possible values of one variable and the columns representing possible values of the other. The marginal distribution of one of the categorical variables in a two-way table of counts is the distribution of values of that variable among all the individuals described by the table. (Use the totals in the margins to calculate this). Example: A sample of 200 children from the United Kingdom aged 9-17 was selected from the CensusAtSchool website. The gender of each child was recorded along with which superpower they would most like to have. Here are the results. Calculate the marginal distribution of superpower preferences. Make a graph to display the marginal distribution. Describe what you see. Gender Superpower Female Male Total Invisibility Superstrength Telepathy Fly Freeze Time Total To examine the relationships between variables, we need to calculate some well-chosen percentages from the counts in the table. A conditional distribution of a variable describes the values of that variable among individuals with one specific value of another variable. There is a separate conditional distribution for each value of the other variable. For example, the conditional distribution of superpower preferences for females would tell us what percentage of females prefers each superpower. Example: Using the data above, calculate the conditional distribution of superpower preference for each gender. (This means figure out what percentage of girls want each superpower and what percentage of boys want each superpower.)

7 To compare the conditional distributions of a categorical variable, we use side-by-side bar graphs (or comparative bar graphs). These use the same horizontal and vertical axes for the bar graphs of two of more groups. The bar charts are usually color-coded to indicate which bars correspond to each group. Relative frequency should be used for the vertical axis. If specific values of one variable tend to occur in common with specific values of the other, we say there is an association (or relationship) between the two variables. For example, if males prefer certain superpowers more than females (or vice-versa), we would say that there is an association between gender and superpower choice. Do not use the word correlation when you mean association. Correlation has a very specific meaning in statistics, which we will talk about later in the year. Example: Draw a side-by-side bar graph comparing the superpower choices of males and females. Use percentages for the vertical axis. Then draw a segmented bar graph for each gender. Describe what you see. Does there appear to be an association between gender and superpower choice?

8 Displaying Quantitative Data with Graphs One of the most common parts of a statistical problem is finding an appropriate way to display data. Quantitative data can t be displayed the same way as categorical data (bar graphs and pie charts don t work). The most common ways to display quantitative data are dotplots, stemplots, histograms, and boxplots. How to Examine the Distribution of a Quantitative Variable Describe the overall pattern of a distribution by describing its shape, center, and spread. Point out any outliers (unusually small or unusually large data values). Always put your descriptions in context! Describing Shape: How many peaks does the distribution have? Don t count minor ups and downs, only major peaks. o Unimodal: One peak. o Bimodal: Two peaks. o Multimodal: Three or more peaks. Is the distribution symmetric or skewed? o If the right and left sides of the graph are close to mirror images of each other, describe the distribution as approximately symmetric. Always use the words approximately or roughly, because in real life, distributions of data are almost never perfectly symmetric. o If the right side of the graph is much longer than the left side (tail to the right), describe the distribution as skewed to the right or skewed to positive values or positively skewed. o If the left side of the graph is much longer than the right side (tail to the left), describe the distribution as skewed to the left or skewed to negative values or negatively skewed. Describing Center: Use the median or the mean. Describing Spread: Use the range, interquartile range, or standard deviation, or say something like, The data vary from a low of to a high of.

9 Dotplots: 1. Draw a horizontal line and mark it with an appropriate measurement scale. 2. Locate each value in the data set along the measurement scale and represent it by a dot above the line. If there are two or more observations with the same value, stack the dots vertically. To compare two distributions, stack the dotplots on top of each other, using the same scales. Make sure to label the two groups being compared. Example: Here are the hair lengths of 26 AP Statistics students, in inches. Make a dotplot. Describe the distribution Here are the lengths sorted by gender. Make parallel dotplots of the hair lengths for males and females. Comment on what you see. Male: Female: Stemplots (or Stem-and-Leaf Plots): Each number in the data set is broken into two pieces a stem and a leaf. The stem is the first part of the number and consists of the beginning digits. The leaf is the last part of the number and consists of the final digit(s). 1. Choose stems (one or more of the leading digits) that divide the data into a reasonable number of groups (at least 5, but not too many). List possible stem values (not just those that actually appear in the data set don t skip stems) in a vertical column. Draw a vertical line to the right of the stems. 2. The next digit(s) after the stem become(s) the leaf. List the leaf for every observation to the right of the corresponding stem. 3. Include a key explaining what the stems and leaves represent, e.g., 2 5 represents 2.5 seconds

10 It is common to round and/or truncate (leave off) the remaining digits. For example, in a stemplot of annual salary, we might represent $35,360 as 35 3, 35 4, or as 3 5, depending on our data set. If necessary, consider using split stems. Write each stem more than once, and assign the lower group of leaves to the first stem and the higher group of leaves to the next. For example, put the leaves 0-4 with the first stem and the leaves 5-9 with the second. If you do this, be sure that each stem is assigned an equal number of possible leaf digits (two stems, with five possible leaves each; or five stems, with two possible leaves each). To compare two groups, make a back-to-back stemplot. Use the same set of stems and write the leaves for one group to the right and for the other group to the left. Be sure to label each side to indicate which group is being represented. Example: The data below shows the number of pairs of shoes owned for male and female AP Statistics students. Make a back-to-back stemplot of the data using split stems. Comment on the main differences between the two data sets. Female Male

11 Histograms: 1. Divide the range of the data into intervals of equal width. The intervals are called bins. The low value in each bin is included in the bin, but the high value is not. For example, the bins might be 0 to < 3, 3 to < 6, 6 to < 9, etc. If the data are discrete (the observations take only whole number values) and are tightly packed, the bins are usually centered at the integer values with a width of one unit, so the rectangle for 1 is centered at 1 (0.5 to < 1.5), the rectangle for 2 is centered at 2 (1.5 to < 2.5), etc. There are no set-in-stone rules for how many bins to use (5 to 10 is a common number), but it may be a good idea to see what the graph looks like with different width bins. It can change quite a bit! 2. Find the count (frequency) or percent (relative frequency) of individuals in each class. 3. Label and scale your axes. Mark the boundaries of each interval on a horizontal axis and use either frequency or relative frequency on the vertical axis. 4. Draw a rectangle for each interval. The edges of the rectangle are at the interval boundaries, and the height of the rectangle represents the frequency or relative frequency of the individuals in each interval. Histograms and bar graphs are different! o Bar graphs are used for categorical data. Histograms are used for quantitative data. o With histograms, you have one variable, and you sort the values of the variable by placing them into bins. The histogram shows how many measurements are in each bin. The rectangles can t be rearranged. o With bar graphs, the bars represent different categories. The height compares some characteristic of each category. The bars can be rearranged. o The bars in bar graphs are generally unconnected. The bars in histograms are connected. Example: The following data gives the average points scored per game (PTSG) for the 30 NBA teams in the regular season. Draw two relative frequency histograms using different bin widths. Describe the distribution

12 Describing Quantitative Data with Numbers Population: The entire collection of individuals or objects that you want to learn about. Sample: A part of the population that is selected for study. Resistant Measure: A measure that is not influenced very much by strong skewness or extreme values. Measures of Center: The most common measures of center are the mean and the median. Mean: The sum of the values divided by the number of observations. x1+ x xn xi If the n observations in a sample are x1, x2,..., x n, the mean is x= =. n n The mean can be thought of as the average value, the fair share value, or the balance point of a distribution. The mean is not a resistant measure. It is very sensitive to outliers and skewness. The mean of a sample is abbreviated x (pronounced x-bar ) and the mean of a population is abbreviated μ x (the Greek letter mu, pronounced myoo ). They are both calculated the same way. The distinction will be important later in the year. If the problem doesn t specify whether the data represent a population or a sample, assume you are dealing with a sample and use x. Median (M): The midpoint of a distribution. Half of the observations are smaller than the median and half of the values are larger than the median. To find the median: 1. Put the n observations in order from smallest to largest. 2. If the number of observations, n, is odd, the median is the middle observation of the ordered list. 3. If the number of observations, n, is even, the median is the average (mean) of the two middle observations in the ordered list. The median can be thought of as the typical value of a variable. The median is a resistant measure. It is not changed greatly by strong skewness or outliers. Comparing the Mean and the Median: The mean and median of a roughly symmetric distribution are close together. If the distribution is exactly symmetric, they are equal. However, outliers and other extreme values drag the mean toward them without having much effect on the median. As a result, in skewed distributions, the mean will be further out in the long tail than is the median.

13 Example: Here are the amounts of fat (in grams) in McDonald s beef sandwiches. Make a stemplot of the distribution and comment on its shape. Then calculate the mean and the median amount of fat. Sandwich Fat (g) Sandwich Fat (g) Hamburger 9 Big N Tasty 24 Cheeseburger 12 Big N Tasty with Cheese 28 Double Cheeseburger 23 McRib 26 McDouble 19 Mac Snack Wrap 19 Quarter Pounder 19 Angus Bacon & Cheese 39 Quarter Pounder with Cheese 26 Angus Deluxe 39 Double Quarter Pounder with Cheese 42 Angus Mushroom & Swiss 40 Big Mac 29 Example: Forty students were enrolled in a statistical reasoning course at a California college. The instructor made course materials, grades, and lecture notes available to students on a class web site, and course management software kept track of how often each student accessed any of these web pages. One month after the course began, the instructor requested a report of how many times each student had accessed a class web page. The 40 observations are below. Wasn t it nice of me to put them in order? (not a typo) Here is a dotplot of the data. Describe the distribution. Based on the graph, do you expect the mean or the median to be higher? Calculate the mean and the median to see if you were right. Which measure would be the best choice to describe center in this situation? Number of Visits to Class Website

14 Measures of Spread: Numbers that describe how spread out the data are. The most common are the range, the interquartile range, and the standard deviation. Range: The difference between the maximum and minimum values. Interquartile Range (IQR): First, calculate the quartiles: 1. Arrange the data in increasing order and locate the median, M. (The median is sometimes called the second quartile, or Q2). 2. The first quartile (Q1) is the median of all the observations lower than the median. 3. The third quartile (Q3) is the median of all the observations higher than the median. The interquartile range is calculated as follows: IQR = Q3 Q1 The IQR is the range of the middle 50% of the data. The range and interquartile range are numbers! Don t say The range is 5 to 30. In that case, the range would be IQR Rule for Outliers: Any observation that falls more than 1.5 IQR above the third quartile or below the first quartile. Always check for outliers and examine them closely! They may be errors, or they may tell you something important about your data that you need to pay attention to. Don t ignore them. Boxplots (or Box and Whisker Plots): 1. Find the Five-Number Summary: Minimum Q1 M Q3 Maximum 2. Check for outliers. You must always show this step. Calculate the IQR. Find Q1 ( 1.5 IQR) and Q3+ ( 1.5 IQR). If you have any data points outside these thresholds, they are outliers. 3. Draw the boxplot: Draw a central box from Q1 to Q3. Draw a vertical line in the box to mark the median. Draw the whiskers : lines extending from the box out to the smallest and largest observations that are not outliers. Mark outliers with dots in the appropriate locations. Each section of a boxplot contains 25% of the data. The lower quartile is higher than 25% of the data. The median (or second quartile) is higher than 50% of the data. The upper quartile is higher than 75% of the data. Boxplots are useful for comparing distributions, but you have to be careful with them. They sometimes mask important features of a distribution. For instance, you can t tell from a boxplot whether the distribution is unimodal or bimodal.

15 Example: The data below shows the number of text messages sent by a random sample of students in a day. Draw parallel boxplots of the number of texts sent for male and female students. You must show how you determined whether there are outliers. Compare the distributions. What conclusions can you draw about the texting habits of males and females? Male Female Standard Deviation: The most common measure of spread is the standard deviation. It measures the average or typical distance of the observations from the mean. The formula for standard deviation is slightly different depending on whether you have all the data for the entire population or are dealing with a sample from the population. For a Sample: If the n observations in a sample are x1, x2,..., x n, and the mean is x, the standard deviation is given by: s The sample standard deviation is abbreviated x. variance, abbreviated s 2 x. x ( x x ) + ( x x ) + + ( x x ) ( x x ) n i = = n 1 n 1 s The square of the standard deviation is called the

16 For the Population: The standard deviation of a population of size N with mean μ and observations x1, x2,..., x n is given by: ( x μ) + ( x μ) + + ( x μ) ( x μ) n i σ x = = N N The population standard deviation is abbreviated σ x (the Greek letter sigma). The population variance is abbreviated σ 2 x. The reason that we divide by n 1 in a sample is complicated. We ll discuss it later in the year. Always use s x rather than σ x unless you know that the data represent the entire population, which is rare! Calculating the standard deviation by hand: 1. Calculate the mean, x. 2. Find the distance of each observation from the mean (the deviations). 3. Square each of these distances to eliminate negative numbers. 4. Average the squared distances by adding them together and dividing by n 1. This gives the variance, s 2. x 5. Take the square root of the variance to get the standard deviation, s x. 6. Interpret your result. The standard deviation is the average or typical distance of the observations from the mean. Example: The table below shows the sugar content in several types of candy bar. Find the mean and standard deviation of the data. Interpret your result in context. Candy Bar Sugar (grams) Hershey s Milk Chocolate 31 Krackel 24 Kit Kat 22 York Peppermint Pattie 25 Reese s Peanut Butter Cups 25 Snickers 30 Milky Way 35 Twix 27 3 Musketeers 40 Baby Ruth 33 Nestle Crunch 24 Butterfinger 29 Mounds 21 Almond Joy 20 Mr. Goodbar 22 Payday 21 Heath 23

17 Example: Each of these distributions has a mean of 5. Rank the standard deviations from lowest to highest Properties of the Standard Deviation The standard deviation measures the spread about the mean and should only be used when the mean is chosen as the measure of center. The standard deviation is always greater than or equal to zero. If there is no variability (all observations have the same value), the standard deviation is zero. Otherwise, it is greater than zero. The standard deviation has the same units of measurement as the original observations. This is one reason we often prefer the standard deviation over the variance. The standard deviation is not a resistant measure. A few outliers can change its value dramatically. Choosing Measures of Center and Spread: Use the median and IQR for describing a skewed distribution or a distribution with strong outliers. Use the mean and standard deviation for describing reasonably symmetric distributions without outliers. ALWAYS GRAPH YOUR DATA! Numerical measures of center and spread report specific facts about a distribution, but don t give information about its entire shape. You may miss something important if you don t graph the data.