STAT 200 Chapter 1 Looking at Data - Distributions

Transcription

1 STAT 200 Chapter 1 Looking at Data - Distributions What is Statistics? Statistics is a science that involves the design of studies, data collection, summarizing and analyzing the data, interpreting the results and drawing conclusions. Inferences (conclusions) are made about specific random phenomena on the basis of relatively limited sample material. Data and Variables Let s look at the data extracted from medical records of 50 patients with low back pain: Subject Age Gender In employment? Duration Severity First episode of pain of pain of pain 1 35 F No 3 weeks mild < 1 year 2 42 F Yes 13 weeks severe 1-6 years 3 21 M Yes 4 weeks moderate < 1 year 4 59 F No 72 weeks moderate 11 years M Yes 30 weeks severe 6-11 years Each row contains information on a case that can be an individual or an experimental unit. A variable refers to a characteristic of interest, e.g. age and gender. A variable can be: 1. qualitative/categorical 2. quantitative (measured on a numerical scale) (a) discrete (integer-valued) (b) continuous (assumes any value on the number line) 1

2 For the above data set, a case is: variable variable type discrete or continuous? unit Age Gender In employment? Duration of pain Severity of pain First episode of pain HOW TO SUMMARIZE A DATA SET? - by means of graphical displays and descriptive statistics. Displaying for Categorical Data (Section 1.1) For categorical data, the key is to group similar things together. I. Frequency tables / Relative frequency tables A (relative) frequency table shows all the categories of a categorical variable together with their (relative) frequencies. The relative frequency is the frequency expressed in percentages. For non-overlapping categories, their percentages should add up to 100%. 2

3 e.g. City of residence of past 30-day transit riders in the GVRD (year 2004) II. Bar Charts A bar chart shows rectangular bars each representing a category. The bars have the same width, and their heights represent the frequency or relative frequency. 3

4 III. Pie Charts A pie chart shows categories as slices in a circle. The area of each slice is proportional to the fraction of the whole for the category it represents. Displaying for Quantitative Data (Section 1.1) I. Frequency distribution Frequency distribution divides the data into classes and counts the number of occurrences (frequency) in each class. Constructing a frequency distribution: 1. Identify the smallest and largest measurements 2. Divide the interval between the smallest and largest measurements into non-overlapping subintervals (classes) Note that each measurement should fall into one and only one subinterval (The left end-point convention)! Also, too many or too few class intervals should be avoided. General rule: number of classes n where n is the number of observations. 3. Find the number of observations, frequency f i, in each class interval 4

5 4. Compute relative frequency, f i, for each class interval n 5. Compute cumulative frequency Example: Age data of 25 patients: Class Frequency Relative Cumulative Class Density interval (f i ) frequency ( f i ) n frequency width What percentage of the patients are younger than 50? What percentage are between 48 and 65? Above what age do the oldest 20% of the patients fall? II. Histogram Constructing a histogram: 1-5. Follow steps 1 to 5 for constructing frequency distribution 6. Find the width of each interval and compute the density Relative frequency Density = class width 7. Draw rectangular boxes of height=density (one rectangle for each class); the data are plotted along the x-axis, and the density is plotted along the y-axis Note: 1. When the class widths are all equal, it is okay to plot the frequency or relative frequency on the y-axis. Such a histogram is called the frequency or relative frequency histogram. When the class widths are unequal, one should plot the density. We call this a density histogram. 5

6 2. The total area under a density histogram = 1. Total area = sum of (class width density) over all the classes Shape of distribution of data unimodal, bimodal or multimodal? symmetric or skewed? skewed to the right (positively skewed; long right-hand tail) skewed to the left (negatively skewed; long left-hand tail) presence of outliers, i.e., unusually small or large observations? III. Stemplot Constructing a stemplot: 1. Partition each observation into a stem and a leaf; the stem corresponds to one or more leading digits, and the leaf corresponds to the single trailing digit 2. Write the stems in a vertical column 3. Record the leaf for each observation in the row corresponding to its stem Numerical Description of Quantitative Data (Section 1.2) Measures of Center The center of a data set describes where the data tend to cluster or center about. I. Mean It is the arithmetic average of the observations (x i s) # observations n = n i=1 mean ( x) = x i n = x 1+x 2 + +x n n 6

7 II. Median It is the middle number of the observations (x i s). It divides the data set into two equal parts. Calculating the median: 1. Rank the data in ascending order 2. If the number of observations (n) is odd: median = ( n+1 )th observation 2 even: median = average of the ( n)th and ( n + 1)th observations 2 2 Measures of Center and Shape of distribution symmetric: mean median (mean = median only when the distribution is perfectly symmetric) skewed to the right: median<mean skewed to the left: median>mean If the data distribution is (roughly) symmetric, both the mean and median are good measures of center of the data. However, if the distribution is skewed, the median will be a better measure. Why? Measures of Spread How are data scattered around the center of a distribution? We look at the spread (variability or dispersion) of the data. I. Range Range = maximum (largest observation) minimum (smallest observation) 7

8 II. Variance and Standard Deviation Variance is the arithmetic average of the squared deviation from the mean. # observations = n n variance (s 2 i=1 ) = (x i x) 2 n 1 = (x 1 x) 2 + (x 2 x) (x n x) 2 n 1 Standard deviation (SD) is the square root of the variance SD = s = s 2 (of the same unit as the observations) Why do we take the squares of the deviations from the mean but not the actual deviations from the mean? Properties of the variance and the SD: non-negative increase with the variability of the data equal to 0 if the observations in the data set are all equal SD measures the typical distance of an observation from the mean III. Interquartile range (IQR) It is the range that encloses the middle 50% of the observations from the distribution. Definition: The pth percentile (0 < p < 100) In a data set with n observations, the pth percentile refers to the number such that p% of the observations fall below it, and (100 p)% fall above it. Definition: Quartiles (Q 1, Q 2, Q 3 ) The three quartiles divide the data into 4 equal parts: Q 1 = first quartile = 25th percentile Q 2 = second quartile = 50th percentile = median Q 3 = third quartile = 75th percentile IQR = Q 3 Q 1 8

9 The 5-Number Summary & Box-plots The 5-number summary include minimum, Q 1, Q 2, Q 3, maximum. The box-plot provides a graphical display of the 5-number summary, and is useful for making comparisons between two or more distributions. Observations whose values exceed Q IQR or fall below Q1 1.5 IQR are suspected outliers and are plotted as separate dots in a boxplot. An example: Sensitivity to Outliers A summary statistic is said to be sensitive to outliers if its value is easily influenced in the presence of outliers. Sensitive to outliers mean range,variance,sd Not sensitive to outliers median IQR 9

10 When to use which summary statistics? Note: If the data distribution is (roughly) symmetric, the mean and median, variance, SD and IQR provide good summaries for the data. If the distribution is skewed or when there are outliers, the median and the IQR will be better measures. The median is always reported along with the IQR; the mean with the variance or the SD. If there are outliers in the data, do not simply discard them without justification. Effect of a linear transformation Suppose we have a data set with observations x 1, x 2, x 3,..., x n. 1. Adding a constant add a constant a to each observation in the data any measure of center (mean and median) will be shifted by the constant a shifting the data does not change the spread; any measure of spread (variance, SD, range, IQR) will remain unchanged 2. Multiplying a constant multiply each observation in the data by a positive constant b both the center and spread will change measures of center (mean and median), and measures of spread (SD, IQR, range) will be multiplied by the constant b the variance of the new data will be b 2 times the original variance 3. Linear transformation: x new = a + bx for a measure of center c, c new = a + b c for a measure of spread d (except variance), d new = b d for variance, s 2 new = b 2 s 2 10

11 Example: SAT and ACT scores SAT and ACT are standardized tests used in the college admissions process. The two exams use very different scales and hence comparisons of performance are difficult. A convenient rule of thumb is SAT = 40 ACT Given that the ACT scores of 2500 students have the following summary statistics: minimum=19, mean=27, SD=3, median=28, Q3=30, IQR=6 Find the summaries of the equivalent SAT scores. Density Curves (Section 1.3) a curve that approximates the shape of a density histogram always above the horizontal axis the total area under a density curve is 1 the area under a density curve over the range [a,b] represents the percentage of observations falling in that range. The Normal Distribution (Section 1.3) Many phenomena give rise to data whose distribution is bell-shaped and roughly symmetric. Examples of such data include the birth weight of babies, the pulse rate per minute of adults, the time required for an AirCare technician to complete an inspection test for a vehicle. Note that the birth weight, the pulse rate and the time for completion in the examples are quantitative variables. An important distribution that governs such distributions is the normal distribution. Questions of interest: 1. What is the fraction of new born babies in Vancouver that are heavier than 5.0kg? 2. What is the proportion of adults whose pulse rates are between 72beats/min and 90beats/min? 3. What is the percentage of time an AirCare inspection test is done in 10 minutes? Characteristics of a normal distribution: bell-shaped; unimodal 11

12 perfectly symmetric about the mean µ the spread of the distribution is determined by the value of the standard deviation σ the normal distribution is denoted by N(µ, σ) for a variable X following the normal distribution (we write X N(µ, σ)), the density function f(x) is given by f(x) = 1 σ (x µ) 2 2π e 2σ 2 the total area under the density curve f(x) is always equal to 1, regardless of the values of µ and σ. To determine the adequacy of the normal distribution in describing the distribution of a continuous variable, compare the density histogram of a large sample of the data to the normal density function. Check if the shape of the density histogram of the data can be well approximated by the normal density curve. The Rule Interval % of observations falling within the interval within 1 σ of µ about 68% within 2 σ of µ about 95% within 3 σ of µ about 99.7% The probability that X will have a value falling in the interval [a,b] (a, b are constants), P (a X b), is the area under the density curve over the interval [a,b]. The area can be obtained by integrating the density function over [a,b] with respect to x. The z-score the z-score for an observation x is defined as the distance between x and the mean µ, divided by the standard deviation σ, as given below: z = x µ σ Thus, the z-score gives the distance between an observation and the mean in units equal to the standard deviation. When x = µ, z = 0. A positive (negative) z-score implies the observation has a value above (below) the mean. 12

13 For a variable X N(µ, σ), the corresponding z-score Z = X µ follows the σ standard normal distribution with mean 0 and standard deviation 1. The standard normal distribution is used for probability calculations. Determining probabilities for normal variables 1. Sketch a normal curve and label the mean. Then shade the area that corresponds to the probability of interest. 2. Convert the x-values at the boundaries of the area to z-scores using the formula z = x µ σ The conversion process is called standardization. 3. P (a X b) = P ( a µ X µ b µ ) = P ( a µ Z b µ ) σ σ σ σ σ 4. Use Table A in the textbook to look up the areas. Examples (for Section 1.3) 1. Find the following areas with the numbers as z-scores: (a) smaller than (b) bigger than 2.15 (c) between and 1.65 (d) between 0.20 and 2.80 (e) smaller than Find the z-scores corresponding to the following percentiles: (a) 80 th (b) 25 th 3. Suppose the birth weight of babies born in Vancouver follows the normal distribution with mean 3.5kg and standard deviation 0.7kg. (a) What is the probability of a randomly selected baby weighing more than 3.0kg at birth? (b) What is the probability of a randomly selected baby weighing between 2.8kg and 4.2kg at birth? 13

14 (c) Find the 40 th percentile of the birth weights among all babies born in Vancouver. (d) How heavy a randomly selected baby must be in order to be in the heaviest 20% of all the babies born in Vancouver? (e) Within what range do the middle 95% of the birth weights fall? 4. Suppose the length of time required to assemble a photoelectric cell is normally distributed with mean µ and a standard deviation of 1.3 minutes. It is known that the probability that a randomly selected photoelectric cell requires more than 20 minutes to assemble is 0.3. Find the value of µ of the normal distribution. 14