Beyond classical statistics: Different approaches to evaluating marking reliability

Transcription

1 Beyond classical statistics: Different approaches to evaluating marking reliability E. Harrison, W. Pointer, Y. Bimpeh & B. Smith

2 What do we mean by marking reliability? Reliability = when something can be repeated many times and produces the same results Marking reliability = when marking of a candidate response can be repeated many times and results in the same (or similar) mark(s) Intra-marker reliability (within markers) Inter-marker reliability (between markers)

3 Why does reliable marking matter? awarded score = true score + error Affects how certain we can be that a candidate has been awarded their true score Implications for (grade) classification accuracy Can be the difference between University/not, or job/not Grade changes undermine confidence in the system Ultimately, inaccurate grading upsets the rank order of candidates and is thus unfair

4 How can we measure marking reliability? Variety of methods for monitoring marker consistency across marking: Classical test theory Generalisability theory (Brennan, 2001) Many-facet Rasch model (Engelhard, 1996; Myford and Wolfe, 2003, 2004) Hierarchical rater modelling (Patz, 1996) Rater bundle model (Wilson and Hoskens, 2001) Signal detection rater model (DeCarlo, Kim, and Johnson, 2011) Latent trait model (Wolfe and McVay, 2012) Confirmatory factor analysis (Bollen, 1989; Green & Yang, 2009; Jöreskog, 1971)

5 Example of seeding data Candidate responses (seeds) markers x x x x x x 2 x x x x x x x x x 3 x x x x x x 4 x x 5 x x x x x x x x x 6 x x 7 x x x x x x x x x 8 x x x x 9 x x x x x 10 x x x x x

6 A G theory approach to marking reliability Liz Harrison

7 Background: Classical test theory (CTT) Observed score on this occasion X po = T p +e po Error on this occasion True score Reliability: the correlation between observed and true Cronbach s α often used as a proxy, ideally >0.8 Error undifferentiated

8 Generalisability theory Cronbach and others developed G theory by generalising some of the basic assumptions of CTT Sources of variation are referred to as facets e.g. person, marker, item, form, time, setting, school, etc. A test is viewed as a sample from a wider universe of admissible observations on each facet Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag

9 The G study and the D study G study: Researcher must decide which facets are sources of error Carefully designed to estimate as many sources of error as is reasonably and economically feasible D study: The results from the G study can be used to inform a decision: a D study

10 Component variance approach Person x marker for a single item: Person effect Marker effect Interaction effect + residual X pm = μ + (μ p μ) + (μ m μ) +(X pm μ p μ m + μ) σσ 2 (XX pppp ) = σσ 2 (pp) + σσ 2 (mm) + σσ 2 (pppp, ee)

11 Person x marker x item design X pmi = μ σσ 2 XXXXXXXX = σσ 2 pp + σσ 2 mm + σσ 2 ii + σσ 2 ppmm + σσ 2 ppii + σσ 2 mmii + σσ 2 (pppppp, ee) grand mean + μ p μ person effect + μ m μ marker effect + μ i μ item effect + μ pm μ p μ m + μ person x marker effect + μ pi μ p μ i + μ person x item effect + μ mi μ m μ i + μ marker x item effect + X pmi μ pm μ pi μ mi + μ p + μ m + μ i μ person x marker x item + residual

12 Two measurements of error (person x marker design) σ : absolute error (inter-marker agreement) σσ 2 = σσ2 mm nn mm + σσ2 pppp, ee nn mm σ δδ : relative error (inter-marker reliability) σσ 2 δδ = σσ2 pppp, ee nn mm nn mm = number of markers in the D study

13 Generalisability Coefficient: reliability Eρ 2 = σσ 2 (pp) σσ 2 (pp) + σσ 2 (δδ) Eρ 2 correlation(x p, X pm ) 2 Universe score

14 Design issues Crossed Nested Fixed effects Unbalanced Analogous ANOVA procedure. Henderson s Method 1 (Biometrics, 1953)

15 The D study for mark-remark data Use the simple person x marker design per item We are interested in the impact of error on an individual student; marked by one marker Absolute error: Reliability: σσ ii 2 = σσ ii 2 (mm) + σσ ii 2 (pppp, ee) EEρρ 2 (ii) = σσ ii 2 (pp) σσ ii 2 (pp) + σσ ii 2 mm CC pppp + σσ ii 2 (pppp, ee) See Brennan p.236

16 D study results at item level Item Tariff %σ(p) %σ(m) %σ(pm,e) σ i (Δ) 1a b c d e a b bE c cE

17 Other D studies Eρ i 2 Item absolute error Item Tariff n m =1 n m =2 n m =3 n m =5 n m =1 n m =2 n m =3 n m =5 1a b c d e a b bE c cE

18 Conclusion G theory powerful tool to look at marking reliability Can be used to design better assessments Use for continuous improvement of exam papers by flagging items that have been challenging to mark Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag

19 The Many-Facet Rasch Measurement Model William Pointer, AQA

20 The Many-Facets Rasch Measurement (MFRM) Model loooo PP nnnnnnnn PP nnnnnnnn 11 = BB nn DD ii CC jj FF kk Where: PP nnnnnnnn = probability of candidate n being rated k on item i by marker j PP nnnnnnnn 1 = probability of candidate n being rated k - 1 on item i by marker j BB nn = ability of candidate n DD ii = difficulty of item i CC jj = severity of marker j FF kk = difficulty of scale category k relative to scale category k - 1

21 Missing Data There are often gaps in the data: Not all candidates will answer every item Not all markers will mark every candidate Not a problem for Rasch models The only requirement is that the data is fully connected

22 Key Statistical Indicators A fixed chi-square test Separation Strata Separation reliability

23 Key Statistical Indicators Item Chisquare d.f. Significance Separation Strata Reliability 1a < b < c < d < e < a <

24 Types of Marker Effects Leniency / severity Randomness Central tendency Halo effect Differential leniency / severity

25 Leniency / Severity Are some examiners more severe or lenient than others? Item 1c Chi-square D.F. 19 Significance <0.01 Separation 2.76 Strata 4.01 Reliability 0.88

26 Leniency / Severity Fair average average mark awarded by a marker adjusted to take into count the ability of the candidates that they have marked

27 Randomness Are markers applying the scale in a consistent manner? Key statistical indicators: Infit/outfit SR/ROR correlations

28 Randomness Infit Outfit Correlation MnSq ZStd MnSq ZStd PtMea PtExp Examiner

29 Central Tendency Do markers tend to overuse the middle categories?

30 Summary The model performs well even with lots of missing data The model allows us to analyse candidates, items and markers in detail Person ability estimates are not dependent on markers Can diagnose various different types of marker effects

31 References Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, Linacre, M. (1989). Many-facet Rasch measurement. Chicago: MESA Press. Masters, G. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, Myford, C. & Wolfe, E. (2003). Detecting and Measuring Rater Effects Using Many- Facet Rasch Measurement: Part I. Journal of applied measurement, Myford, C. & Wolfe, E. (2004). Detecting and Measuring Rater Effects Using Many- Facet Rasch Measurement: Part II. Journal of applied measurement, Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copyright AQA and its licensors. All rights reserved. AQA Education (AQA) is a registered charity (registered charity number ) and a company limited by guarantee registered in England and Wales (company number ). Registered address: AQA, Devas Street, Manchester M15 6EX.

32 Marking Reliability: using confirmatory factor analysis Yaw Bimpeh

33 Background Data from routine monitoring of on-screen marking has the potential to be used for regular measurement of marking reliability by the UK awarding bodies The objective of the current research is to look at the Confirmatory factor analytic (CFA)-based perspectives on marking reliability

34 Why CFA approach? Many facet Rash model provides micro level of marker analysis G-theory provides macro level of marker analysis CFA provides both micro and macro level of marker analysis

35 CFA approach CFA provides the researcher with a lens to see both the tree and forest of marking

36 Confirmatory factor analytic perspectives on reliability Joreskog (1971) originally formulated the congeneric test model upon which CFA-based estimates of reliability are grounded; it is based on a set of replicates define along a single facet of measurement (e.g., markers, items), and that facet is assumed to be fully crossed with the objects of measurement (e.g., persons)

37 Analysing marking data using congeneric measurement model Xi is marker i score λλ ii is the factor loading ξ is the true or latent score E is the measurement error Congeneric indicators are ones that measure the same latent score There are no restrictions on the factor loadings or error variances except that the error variances are independent

38 Estimation of reliability for general congeneric measures Wright (1918, 1934) developed a method of estimating causal path coefficients by decomposing the correlations among a set of variables Using Wright s tracing rules, one can obtain the reliability of X (i.e., the sum of X1,X2,...Xk) for congeneric indicators of markers. We do this by calculating the proportion of variance due to ξ and the proportion due to E. The estimation of reliability for the general case of congeneric measures, is addressed in the framework of covariance structure analysis (e.g., Feldt & Brenan, 1989; Gilmer & Feldt, 1983; Raykov, 2001b) A more efficiently approach is via latent trait modelling (e.g., Jöreskog, 1971; Raykov, 2009; Raykov, Dimitrov, & Asparouhov, 2010).

39 CFA-based perspective on marking reliability For each item i: Inter-marker reliability = KK ii kk=1 λλ kkkk KK ii kk=1 λλ kkkk 2 2 KK + ii kk=1 vvvvvv(eekkkk ) where λλ kkkk is the marker loading vvvvvv(ee kkkk ) is variance due the measurement error of the kth marker (k = 1,,KK ii ) of seed item i KK ii is the number of markers of scoring seed item i This measure of reliability is the so called omega (McDonald, 1999)

40 Handling missing data Missing data is inherent in the seed marking data, since seeds are evaluated by many markers, but each item gets a different candidate-marker combination Little and Rubin (2002) postulate three different types of missing data: Missing Completely at Random (MCAR) Missing at Random (MAR) Missing Not At Random (MNAR) The missing data in marking is either MCAR or MAR The missing data can be handled in CFA using methods such as: Multiple imputation Full information maximum likelihood Traditional methods like pairwise deletion or list-wise deletion

41 Empirical studies of 2016 A-level Dondology marking reliability using CFA method Item M-type Intermarker Reliability Std.err No. of seeds Markers Maximum mark 1c Seeded di Seeded dii Seeded e Seeded a Seeded

42 Results: item 2a Estimate Std.Err z-value P(> z ) factor loading LV =~ A68 (l1) A7485 (l2) A11550 (l3) A13285 (l4) A13826 (l5) A14599 (l6) A18565 (l7) A24085 (l8) A26725 (l9) A26893 (l10) A28889 (l11) A52195 (l12) A57466 (l13) A (l14) A (l15) A (l16) A (l17) A (l18) A (l19) A (l20) A (l21) A (l22) A (l23) A (l24) A (l25) A (l26) A (l27) A (l28) A (l29) A (l30) A (l31) A (l32)

43 Marker Reliability Marker item 1e Grade of item 2a examiner (tariff=6) (tariff=10) A S A S A S A N A N A S A N A N A N A N A N A N A N A S A N A N A N A N A N A N A N A S A N A N A N A N A N A N A N A N A N A N N=normal S=senior

44 Conclusions We provide an alternative method for estimating item level marking reliability The CFA approach provides marker performance information as well as and inter-marker reliability for seed data It provides confidence interval and standard error for the inter-marker reliability estimates It can handle missing data using methods such as: Mean imputation Full information maximum likelihood And traditional methods like pairwise deletion or list-wise deletion It also provides goodness of fit statistics Copyright AQA and its licensors. All rights reserved. AQA Education (AQA) is a registered charity (registered charity number ) and a company limited by guarantee registered in England and Wales (company number ). Registered address: AQA, Devas Street, Manchester M15 6EX.

45 Pros & cons of G-theory Allows delineation of variance components Index values can be compared across disparate items Can be aggregated to unit/subject level Missing data not an issue Not possible to generate detailed marker-level information Problematic with low sample sizes (negative variance issues)

46 Pros & cons of many-facet Rasch models Produces detailed information on markers, candidates and items Specific marker effects can be diagnosed Missing data not an issue Precision of examinees latent scores tends to infinite as number of raters per item increases Potentially biased standard errors Still needs connected data

47 Pros & cons of confirmatory factor analysis Produces model fit estimates Gives both marker performance and overall reliability statistics If poor fit, model cannot be used Computationally more intensive than other methods Not feasible with high missingness (<10% coverage) so double-marked data Requires relatively even # of markers and candidates

48 Discussion Philosophical differences between the models: Raters as rating machines Raters as independent experts G-theory = rating machines Many facet Rasch = independent experts Confirmatory factor analysis = expert machines? Shavelson, R. J., & Webb, N. M. (1991). Generalizability Theory: A Primer. Thousand Oaks, CA: Sage.

49 Any questions? Copyright AQA and its licensors. All rights reserved. AQA Education (AQA) is a registered charity (registered charity number ) and a company limited by guarantee registered in England and Wales (company number ). Registered address: AQA, Devas Street, Manchester M15 6EX.