VI. Linking and Equating: Getting from A to B Unleashing the full power of Rasch models means identifying, perhaps conceiving an important aspect,

Transcription

1 VI. Linking an Equating: Getting from A to B Unleashing the full power of Rasch moels means ientifying, perhaps conceiving an important aspect, efining a useful construct, an calibrating a pool of relevant items that measure it over a meaningful range. So far we have concerne ourselves with processing isolate bunches of items. In this worl, linking an equating item sets has been treate as a istinct an unique phase in the process from conception to measurement. With the technology available, this has typically been the most convenient an efficient approach, an may continue to be so. In the new worl, post fixe-form, paper-base instruments, which are more an more passé, builing a calibrate pool can be an inherent an natural part of the process an not a separate step. Calibration proceures allow us to combine iniviual level test ata across aministrations, perhaps years apart, to check if specific objectivity hols across time or istance. This is just another between-groups comparison, which gives more opportunities for control an investigation of the process. For the moment, we will continue to talk linking an equating the ol fashione way. The terms link an equate are often use interchangeably, sometimes not. In other texts an contexts, the istinction seems to be that equate implies that the tests measure the same construct while linke implies the tests are correlate but may not be synonymous. In this sense, measures from equate tests are interchangeable; measures from linke tests may be useful as valiations, as preictors, or as mileposts but they are not interchangeable. In my worl of Rasch measurement, we consier only tests that measure the same construct. The istinction I will make between link an equate, it is that two forms are: Linke if they are literally connecte through some common, overlapping element, either overlapping items or overlapping examinees or both. Equate if the (logit) scores are on the same scale so that scores from one form are equivalent to an can be compare legitimately to (logit) scores from the other form. Linke implies a irect physical connection between the forms; then an only then, they are equate-able. Equate implies the forms were linke an all the necessary arithmetic has been one. Logit abilities can be estimate from any selection of calibrate items, ieally a unique, carefully tailore set specific for each examinee. An ability estimate from any subset of the calibrate pool can be compare irectly to estimates base on any other subsets taken from the same pool. In that sense, all possible forms that have been or ever coul be built from the pool are equate. All this assumes that the pool conforms to Rasch s requirements; that is, compose of equally vali an reliable items. Because when we starte this by estimating the item ifficulties, we ha one more unknown than we ha equations, we neee to impose a convention. The convention we choose, from among many possibilities, was i = 0. Any collection of items calibrate as a group will be centere at zero: a form for pre-school reaing reainess will be centere at zero; a form for post-grauate stuy of Mile English literature will be centere at zero. The equating problem is to etermine how far it is from the Recognizing Letters zero to the Analyzing Chaucer zero, which we can only solve if the linking has been taken care of.

2 The Conventional Equating App Linking an Equating are converting temperatures on the Kelvin scale to temperatures on the Celsius scale; they use the same units but have ifferent zeros 1. In terms of what points on the scale mean, it oesn t matter which numeric labels we attach; the effect of liqui nitrogen on your skin will be the same whether it s labele 77ºK or -196ºC. We just nee to agree on the scale to communicate with each other. Neither the nitrogen nor your skin much care about the labels. The steps for equating tests or thermometers are: 1. Pick some anchor points. 2. Observe measures for those points on both the new an ol scales. 3. Average each set of measures. 4. Subtract the new average from every point on the new scale. 5. A the ol average to every point on the new scale. This effectively subtracts out the new an as in the ol. Anchor Points Kelvin Celsius Difference boiling point of nitrogen freezing point of salt water freezing point of pure water ~ 273 triple point of water normal human boy temperature boiling point of water at sea level Average If we subtract an a , we have place the Celsius values on the Kelvin scale an one nothing to the Kelvin values. We coul have subtracte the ifference from the Kelvin values an ene up on the Celsius scale just as well. If we ha chosen ifferent anchor points, the means woul be ifferent but not the ifference. Anything we o to the scale values of the anchor points we must o to every point on the scale. The har part is remembering when to a an when to subtract. The process is just as simple though perhaps not as tiy in our worl of mental measurement. Replace the Kelvin values with the logit ifficulties from the previous calibration an the Celsius values with the logit ifficulties for the same items from the current calibration 2. Then the iea is the same an seems too simple to bother talking about. In principle, two forms, like two thermometers, can be linke with one anchor point (one item or one person). Because it is the same item, any ifference in the item s logit ifficulty estimate, beyon ranom error, is ue to a ifference in the arbitrary origins of the calibrations. With an easy form, centere on zero, the link item coul have a positive logit, implying it is harer than the form average. With a ifficult form, also centere on zero, the same link item may have a negative logit, implying it is easier than the average. It oesn t affect how har the item is any 1 Converting Celsius to Fahrenheit, with ifferent units, is a slightly ifferent iscussion. 2 The terms current an previous suggest we are equating this year to last year, which is common. We might also equate the fifth grae scale to the fourth grae scale or this year s test to an existing bank. You might also want to change the row an column labels to something more appropriate.

3 more than using Kelvin or Celsius affects how col nitrogen is. The goal of the equating exercise is to eliminate the ifference ue to the convenient but arbitrary convention of centering on zero by ajusting the logit estimates on one form so that the link item has the same numeric value on both. Assuming our link item has a ifficulty estimate of A when calibrate with form A an an estimate of B when calibrate with form B, then the form B estimate can be mae to equal the form A estimate by subtracting B an aing A. The ajuste ifficulty for the linking item is: 26. * B = (B + t) = B + (A B) = A. So far, this is just as trivial as it seems. The two forms are equate by aing t = A B to every logit on form B. If we o it to one item on form B, we nee to o it to every item on form B to maintain their relative positions. An we can ajust the form B logit abilities in exactly the same way by aing t to each of them without going through the bother of re-computing them from the ajuste ifficulties. In practice, it may not be a goo iea to equate through a single item. With several link items, the equating constant t is simply the ifference between the means of the link items from the two calibrations. 27. t ia ib ia ib ilink ilink ilink ALink BLink, nlink nlink nlink where nlink is the number of link items. The equating constant t is ae to every item an ability on Form B as before. After ajustment, the mean, not each iniviual item, of the link set will be ientical in both contexts. Link Length The number of items that shoul be in the link, like any sample size, epens on how precisely you nee to know the answer. Using the Wright-Douglas approximation, the typical stanar error for t is: 28. se t 1 n Link 6 N A 6 N B, where NA is the number of stuents use in the form A calibration an NB is the number use in the form B calibration. Assuming a moest 500 stuents per form an 15 link items leas to a stanar error for the equating constant of a rather loose 0.04 logits. Doubling the sample size to 1000 stuents, reuces the stanar error by 25% to Turning the relationship aroun provies the link length neee for a given stanar error: 29. n Link 1 se 6 N 6 N 2. t A B

4 If you want to know the equating constant to plus or minus 0.01 logits an have 5,000 stuents per form, a link length of 12 / (5,000 x ) = 24 items is require, which is often pushing the limits of the form builers. Inconveniently, limitations on the test length, content balance, an item exposure often have more to o with the size of the link then with the magnitue of a stanar error neee to make the psychometrician comfortable. An most get very uncomfortable when nlink is as low as 10 items, given the unimpeachable logic of equation 29. Link Control Multiple link items, in aition to increasing the precision of the estimate of the equating constant, are necessary to control the process. Each item pair provies an estimate of the equating constant. Like statistics everywhere, the estimates will not be ientical; we expect some variation an woul be concerne if we in t have it. The problem is to recognize an eliminate outliers from the calculations of the means 3. There are a number of more or less heuristic techniques for ealing with the uncertainty. Figure VI.4 shows the simplest possible link analysis. The points plotte are the logits obtaine from a calibration of the current aministration against the logit ifficulties from the bank. The ata shoul follow a slope one line with the intercepts representing the require equating constant. This example is a very clean link, with one outlier, an an x-intercept of 0.5 (or a little less), ignoring the outlier. Aing 0.5 to every current logit will shift the plotte points vertically so that the slope one line passes (almost) through the origin an the current aministration has been equate to the bank 4. Spotting the outlier in this case is trivial when one visually. Figure VI.4: Sample Link Analysis Simple Graphic Link Analysis Current Logits Bank Logits 3 It shoul also be note that n Link is the number of items we want in the link after the outliers are roppe. 4 This process is symmetrical. The y-intercept, -0.5, coul be ae to the bank logits to shift the plot horizontally an place them on the current scale if that mae sense to anyone.

5 Worst First The same analysis an slightly more precision can be achieve with the appearance of much more rigor if we use tables an numbers. The simplest non-graphical metho is rop the item * with the biggest iscrepancy between the ajuste current logit an bank logit, ti = ia - ib, an to continue oing ropping items until it oesn t have a noticeable effect on the result. Or it may just vacillate up an own epening on whether the last item roppe ha a positive or negative iscrepancy. At that point you are just analyzing noise. One of these situations typically happens with iscrepancies aroun 0.25 logits. The proceure is wiely criticize because it oesn t specify what noticeable effect means or which sie to pick when the result is vacillating. On the other han, some policy makers like this wiggle room. Constant Criterion An approach that at least appears more objective is to choose a criterion logit value an reject * any item from the link if its estimate ti = ia - ib is larger than the criterion in absolute value. The most commonly mentione criterion is 0.3 logits. While rawing this line in the logit san may be more objective, it is just as arbitrary but base on a lot of experience an not much ifferent in practice. This is easy to apply but wiely criticize because it ignores the stanar errors of estimation an applies the same criterion regarless of how well we think we know the item s ifficulty. Stuent s t A little more statistically pure strateg performs a Stuent s t-test on each item using the stanar errors of calibration. Items are rejecte if the t-statistic excees the psychometrician s tolerance level. This is wiely criticize because it consiers the stanar errors an is more tolerant of iscrepancies when they come from poorly estimate items although all have same impact on the equating constant. Robust Z Our final strategy, which seems to incorporate the weaknesses of all of the above, uses a simple robust estimate of the stanar error of the ifferences an applies it uniformly across the items. A robust-z is compute for each link item as: 30. z i ti Q2,.74( Q Q ) where Q1, Q2, an Q3 are the first, secon, an thir quartiles of the istribution of the ti. An item is eligible for rejection when zi is greater than in absolute value. It s a goo thing when the items are very consistent, but in that situation Q3 Q1 is very small so small iscrepancies can make for a large z. An, if we truncate the istribution enough, the quartiles will eventually be very close together. This approach is criticize because it will always fin a biggest iscrepancy base on the empirical istribution an not on any theoretical or philosophical consierations. To provie a rational stopping rule, no (more) items are roppe when: ratio of stanar eviations of the two sets of ifficulties is between 0.9 an 1.1, correlation between the two sets of ifficulties is at least 0.95, an you are in anger of running out of link items.

6 A large z makes an item eligible for ropping but oes not insist that it must go. Some in the business also set a minimum number of link items that they won t go below but sometimes you just have to amit you have a problem. The Arithmetic Table 9 contains a summary of these analyses for the same ata use in Figure 4. The Pool an Current logits are given. The Difference is the Pool Current; the Ajuste is the Current + average Difference; an the Discrepancy is the ifference between the Pool an Ajuste logits. The Stuent s t-statistic is Discrepancy ivie by the stanar error. The Robust Z is efine by equation (30). Table 9: Sample Link Calculations First Roun: All Items Pool ia Current ib Difference ia- ib Ajuste ib+t Discrepancy ia-( ib+t) Stuent s t-statistic Robust Z Mean t = Q2 = St Dev Q3 = Ratio SDs 0.88 Q1 = Correlation 0.94 Secon Roun: Drop Item Mean t = Q2 = St Dev Q3 = Ratio SDs 0.99 Q1 = Correlation 0.99 Using all items, the ratio of stanar eviations is 0.88 an the correlation is Uner Huynh s groun rules, we are force to o something. Only item two is eligible for elimination with a robust z larger than (an a iscrepancy larger than 0.3 logits an a Stuent s t twice as big as anything else.) Dropping this item changes both the SD ratio an the correlation,

7 coinciently, to 0.99, which implies we are finishe. The en result, no matter how we i it, was we roppe one item an ae 0.49 (almost the 0.5 from the plot) to the current logits to place them on the pool logit scale. (The Stuent s t was the weakest test in this situation but the calibration was base on only 100 examinees.) Because all the criteria for a satisfactory link (no iscrepancy larger than 0.3, correlation greater than 0.95, ratio of stanar eviations between 0.9 an 1.1) are met, there is no reason to have compute the robust z statistics for the secon roun. However, having one it, there are two that are larger than This is the nature of this calculation with very consistent items; there will always be a most extreme value. If all the other criteria are met, we en the process, report the scores, an go to inner. Droppe but not Forgotten But before we go, there is an annoying question about what to o with item two. We have exclue it from the link, but o we keep it in the Bank, on the form, an in the examinee scores? If we want to keep it on this form or in the Bank, what logit ifficulty shoul it have? One school of thought is that we shoul continue to use the Bank value because it typically is base on a larger, more efensible sample an analysis. The alternative view is that the value from the current aministration is a better reflection of the current context an if you aren t willing to use that, the item isn t appropriate for this situation. Because the logit ifficulty for that item is not consistent across forms (or aministrations); the item still may have functione acceptably in both situations, albeit ifferently. The first thing to check is that the ifficulties have been matche correctly. If so, it may be the item was unusually amenable to instruction or it may mean it interacte with popular culture (e.g., movies, commercials, music lyrics, current events) in some way others i not. Or it may mean a security breach or other malfeasance. It is generally goo for the psychometrician s peace of min an often informative to ientify the source of any isturbance.