Norm-Referenced vs. Criterion- Referenced Instruments Instrumentation (cont.) October 1, 2007 Note: Measurement Plan Due Next Week All derived scores give meaning to individual scores by comparing them to the scores of a group. The group used to determine derived scores is called the norm group and the instruments that provide such scores are referred to as norm-referenced instruments. An alternative to the use of achievement or performance instruments is to use a criterion-referenced test. This is based on a specific goal or target (criterion) for each learner to achieve. The difference between the two tests is that the criterion referenced tests focus more directly on instruction. Statistics vs. Parameters Descriptive Statistics A parameter is a characteristic of a population. It is a numerical or graphic way to summarize data obtained from the population A statistic is a characteristic of a sample. It is a numerical or graphic way to summarize data obtained from a sample Types of Numerical Data Four Types of Measurement Scales There are two fundamental types of numerical data: 1) Categorical data: obtained by determining the frequency of occurrences in each of several categories 2) Quantitative data: obtained by determining placement on a scale that indicates amount or degree 1
Techniques for Summarizing and Presenting Quantitative Data Summary Measures Visual Frequency Distributions Histograms Stem and Leaf Plots Distribution curves Numerical Central Tendency Variability Central Tendency Arithmetic Mean Median Summary Measures Mode Range Variance Variation Standard Deviation Measures of Central Tendency Central Tendency Average (Mean) Median Mode X = μ = n i= 1 N i= 1 n N X X i i Mean The most common measure of central tendency Affected by extreme values (outliers) 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 12 14 Mean = 5 Mean = 6 Median Robust measure of central tendency Not affected by extreme values 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 12 14 Median = 5 Median = 5 In an Ordered array, median is the middle number If n or N is odd, median is the middle number If n or N is even, median is the average of the two middle numbers Mode A measure of central tendency Value that occurs most often Not affected by extreme values Used for either numerical or categorical data There may may be no mode There may be several modes 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Mode = 9 0 1 2 3 4 5 6 No Mode 2
Variability Refers to the extent to which the scores on a quantitative variable in a distribution are spread out. The range represents the difference between the highest and lowest scores in a distribution. A five number summary reports the lowest, the first quartile, the median, the third quartile, and highest score. Five number summaries are often portrayed graphically by the use of box plots. Variance The Variance, s 2, represents the amount of variability of the data relative to their mean As shown below, the variance is the average of the squared deviations of the observations about their mean 2 ( x ) 2 i x s = n 1 Standard Deviation Calculation of the Variance and Standard Deviation of a Distribution (Definitional formula) Considered the most useful index of variability. It is a single number that represents the spread of a distribution. If a distribution is normal, then the mean plus or minus 3 SD will encompass about 99% of all scores in the distribution. Raw Score Mean X X (X X) 2 85 54 31 961 80 54 26 676 70 54 16 256 60 54 6 36 55 54 1 1 50 54-4 16 45 54-9 81 40 54-14 196 30 54-24 576 25 54-29 841 Variance (SD 2 Σ(X X)2 ) = N-1 = 3640 =404.44 9 Standard deviation (SD) = Σ(X X) 2 N-1 Comparing Standard Deviations Facts about the Normal Distribution Data A 11 12 13 14 15 16 17 18 19 20 21 Data B 11 12 13 14 15 16 17 18 19 20 21 Data C 11 12 13 14 15 16 17 18 19 20 21 Mean = 15.5 S = 3.338 Mean = 15.5 S =.9258 Mean = 15.5 S = 4.57 50% of all the observations fall on each side of the mean. 68% of scores fall within 1 SD of the mean in a normal distribution. 27% of the observations fall between 1 and 2 SD from the mean. 99.7% of all scores fall within 3 SD of the mean. This is often referred to as the 68-95-99.7 rule 3
The Normal Curve Different Distributions Compared Fifty Percent of All Scores in a Normal Curve Fall on Each Side of the Mean Probabilities Under the Normal Curve Interpreting the standard deviation We can compare the standard deviations of different samples to determine which has the greatest dispersion. Example A spelling test given to third-grader children 10, 12, 12, 12, 13, 13, 14 xbar = 12.28 s = 1.25 The same test given to second- through fourthgrade children. 2, 8, 9, 11, 15, 17, 20 xbar = 11.71 s = 6.10 Standard Normal Distribution Z-scores Convert a distribution to: Have a mean = 0 Have standard deviation = 1 However, if the parent distribution is not normal the calculated z-scores will not be normally distributed. 4
The Standard Normal Distribution Why do we calculate z-scores? Z-scores A descriptive statistic that represents the distance between an observed score and the mean relative to the standard deviation xi x z = s x μ z = i σ To compare two different measures e.g., Math score to reading score, weight to height. Area under the curve Can be used to calculate what proportion of scores are between different scores or to calculate what proportion of scores are greater than or less than a particular score. Used to set cut score for screening instruments. From the spelling example A spelling test given to third-grader children 10, 12, 12, 12, 13, 13, 14 xbar = 12.28 s = 1.25 What is the z-score of child who scores 11? What proportion of scores are greater? What proportion of scores are less? What about a child who scores 12? Correlation Correlation Coefficients Positive Correlation Pearson product-moment correlation The relationship between two variables of degree. Positive: As one variable increases (or decreases) so does the other. Negative: As one variable increases the other decreases. Magnitude or strength of relationship -1.00 to +1.00 Correlation does not equate to causation 5
Negative Correlation No Correlation Correlations Negative Correlation Thickness of scatter plot determines strength of correlation, not slope of line. For example see: http://noppa5.pc.helsinki.fi/koe/corr/cor7.html Remember correlation does not equate causation. Validity and Reliability Validity and Reliability Chapter 8 Validity is an important consideration in the choice of an instrument to be used in a research investigation It should measure what it is supposed to measure Researchers want instruments that will allow them to make warranted conclusions about the characteristics of the subjects they study Reliability is another important consideration, since researchers want consistent results from instrumentation Consistency gives researchers confidence that the results actually represent the achievement of the individuals involved 6
Reliability Test-retest reliability Inter-rater reliability Parallel forms reliability Internal consistency (a.k.a. Cronbach s alpha) Validity Face Does it appear to measure what it purports to measure? Content Do the items cover the domain? Construct Does it measure the unobservable attribute that it purports to measure? Criterion Predictive Concurrent Consequential Validity Types of validity (cont.) The scores The construct Here the instrument samples some and only of the construct Types of validity The scores The instrument The construct Here the instrument fails to sample ANY of the construct Here the instrument samples all and more of the construct The scores 7
Perfection! The construct Here the instrument samples some but not all of the construct The construct and the scores! The scores Reliability and Validity 8