1. Exploratory Data Analysis

Transcription

1 1. Exploratory Data Analysis 1.1 Methods of Displaying Data A visual display aids understanding and can highlight features which may be worth exploring more formally. Displays should have impact and be accurate. If you are producing displays by hand, always use a constant scale and a ruler; it is better to use squared paper. Bar Chart Separate bars represent distinct values; bar heights are proportional to frequency. Suitable for all qualitative data and for discrete quantitative data. For unordered qualitative data a bar chart is the only possibility. 8

2 Example Data Set A. Dot Plot Each observation is represented by a dot. Introduced as a graphical method suitable for a teletype. Like a histogram with very narrow intervals. Useful for continuous or large integer data. Heights of students in the class... : :. : : :. : : : :.. : : : : : : : : : : : :.: :: :.:: :.: ::.: :::: : : :.: : > cm 9

3 Stem-and-Leaf Plot The stem reflects the value of the observation "rounded down"; the leaf gives an extra significant figure. Leaves must be of constant width. When putting together a stem-and-leaf plot the leaves generally come out in random order. It is usually helpful to redraw the plot with the leaves in order, as this helps us to construct the box plot. The number of subdivisions should be (very) approximately the square root of the number of observations. Heights of female students in the class

4 Cumulative Frequency Diagram Also called the cumulative frequency ogive, this is an S-shaped (usually) curve which jumps whenever it reaches an observation. The size of the jump is 1/n, or k/n if there are k observations all equal to one another. Histogram A histogram requires grouped data. If you have full numerical values you can do the grouping yourself. It is usually sensible to choose intervals of equal width, like for the stem-and-leaf plot. But sometimes there is no choice, as data arrive already grouped: see later. The rectangles along the horizontal axis are adjacent with no gaps. The area of each rectangle (not the height) is proportional to the frequency (number of observations). The vertical axis is "frequency density". 11

5 Skewness and symmetry Histograms and stem-and-leaf diagrams are good for detecting skewness. A data set which is evenly distributed on both sides of the middles is symmetric; if there is a longer tail to the right, it is right skewed or positively skewed; if the left has the longer tail, it is left or negatively skewed. 1.2 Measures of Central Tendency A measure of central tendency is a single value which is representative (in some way) of the data set as a whole. The mode The mode is the value observed most frequently. For unordered qualitative data the mode is the only representative value. For continuous data there is frequently no mode, or the mode may be meaningless. (See modal intervals, later.) 12

6 The median For an ordered variate, it makes sense to arrange the data set in order and choose the middle one, called the median, to represent the data set. If the sample size, n, is odd, the ½(n+1)th value is the median; if even, the ½nth and the ½(n+2)th have an equal claim. Averaging is not meaningful for qualitative data. The sample mean For quantitative data, denoting the observations by x 1, x 2,, x n, define the sample mean to be the average _ x = x 1 + x x n n Unusually large (or small) values affect the sample mean more than the median. Therefore, on the whole, 13

7 mean > median for right-skewed sample mean < median for left-skewed sample 1.3 Quantiles (Do not apply to unordered qualitative data.) A value u is an upper quartile for the data if at least 75% of the observations are less than or equal to u; at least 25% of the observations are greater than or equal to u. In practice, take the (¾n+½)th value when the data are in order. (Round off quarters.) The upper quartile is also called the 75th percentile. The lower quartile is similarly the 25th percentile, the median the 50th. Also: quintiles (20th, 40th, etc percentiles), deciles (10th, 20th, etc). Quantiles can be read off a cumulative frequency diagram. 14

8 1.4 Measures of spread (Not applicable to qualitative data.) A measure of spread quantifies the extent to which observations differ from the 'representative' value, expressed by a measure of central tendency. Inter-quartile range Associated with the median is the interquartile range, defined as IQR = UQ LQ. Variance and standard deviation The sample variance is denoted s 2 and defined as : S 2 = n 1 1 x 1 x 2 + x 2 x ( x ) 2 } x n This may alternatively be written as 15

9 s 2 = 1 _ { n 1 x x x n n x 2 } The square root of the variance is the standard deviation, s. The standard deviation is the measure of spread associated with the sample mean. If the observations are measured in cm, then the mean and standard deviation are in cm, the variance in cm Box-and-whisker plots The five-number summary of a data set consists of the minimum value, the lower quartile, the median, the upper quartile and the maximum value: the four gaps each contain a quarter of the observations. The box-and-whisker plot, or boxplot, is a graphical representation of this. The box extends from one quartile to the other and is cut by the median. The 16

10 whiskers extend to the minimum and maximum values. Outliers An outlier is any value which does not seem to fit with the rest of the data set. It may be a mistaken observation, it may be due to a known cause, or it may be part of the effect being studied. Outliers distort pictures; it is usually best to exclude them. What constitutes an outlier? Values which are > UQ + 3 * IQR or < LQ 3 * IQR are regarded as definite outliers. Exclude them from the box plot, but list them at the bottom. Observations which are > UQ + 1½ * IQR or < LQ 1½ * IQR are "possible outliers": connect them with dotted lines to the nearest nonoutlier. 17

11 1.6 Visual Comparisons If we wish to compare two or more data sets, some graphical methods are better than others. Bar chart Frequently seen in the media, multiple bar charts should be used with care to avoid confusion. Back-to-back Stem-and-leaf plot A single stem is used, with leaves from one data set going to the right, leaves from the other to the left. Quite good for comparisons. Histograms Up-and-down histograms are seldom used. Other multiple variants are not a good idea. Boxplots Multiple box-and-whisker plots are fine for comparison. Make sure they are 18

12 drawn on a single diagram, to a single scale and are suitably labelled. 1.7 Grouped data Sometimes you do not have individual observations, but only data pre-sorted into groups, which may be of uneven widths. Where the top end of one group does not match the bottom end of the next, it is assumed that rounding has taken place and that the true division point is half way between the interval end points. The recommended forms of display are the histogram and the cumulative frequency diagram. Boxplot is possible, but quantiles must be estimated. Histogram: height = frequency / class width. Cumulative frequency diagram: no jumps; assume observations evenly spread over the interval. No individual mode can be found, but the modal 19

13 interval is the class with the tallest rectangle on the histogram. To calculate sample mean and variance, denote by y i the mid-point of the i-th class, and by f i the number of observations in that class. Then i Sample mean = n 1 n where n = Σf i, and f i y i, Sample variance = 1 n _ n 1 { f i y 2 i n y 2 }. i The quartiles are best found from the cumulative frequency diagram. The place where the c.f. ogive crosses the 25% line can be found using an accurate sketch or, for greater accuracy, linear interpolation. 20

14 1.8 Higher sample moments The sample mean is called the first sample moment; the second sample moment is the sample variance. Higher moments also exist. The third and fourth sample moments are m 3 = m 4 = n 1 { n 1 n 1 { n 1 j j (x i (x i _ x ) 3 }, _ x ) 4 }, From these we can calculate the sample skewness coefficient, equal to m 3 /s 3 (dimensionless), a numerical measure of the lack of symmetry; the sample kurtosis, equal to m 4 /s 4 3 (dimensionless), a numerical measure of the amount of probability contained in the tails. 21