Bayesian Networks in Educational Testing

Size: px

Start display at page:

Download "Bayesian Networks in Educational Testing"

Marian Briggs
5 years ago
Views:

1 Bayesian Networks in Educational Testing Jiří Vomlel Institute of Information Theory and Automation Academy of Science of the Czech Republic This presentation is available at:

2 Contents: Student model and evidence models. Knowledge about a student. Difference between a fixed test and an adaptive test. Optimal and myopically optimal tests. Construction of a myopically optimal fixed test. Test of basic operations with fractions. Results of experiments.

3 Student and evidence models (R. Almond and R. Mislevy, 1999) S 1 S 2 S 3 S 4 S 5 S 1 S 3 S 3 S 4 S 2 S 5 Entropy of probability distribution P (S) H (P (S)) = s P (S = s) log P (S = s) The lower the entropy the more we know about a student.

4 Fixed Test vs. Adaptive Test Q 1 Q 2 Q 5 Q 3 wrong correct Q 4 wrong Q 4 correct wrong Q 8 correct Q 5 Q 6 wrong Q 2 correct wrong Q 7 correct wrong Q 6 correct wrong Q 9 correct Q 7 Q 1 Q 3 Q 6 Q 8 Q 4 Q 7 Q 10 Q 7 Q 8 Q 9 Q 10

5 Entropy in node n H(e n ) = H(P (S e n )) Expected entropy at the end of test t E H (t) = l L(t) P (e l ) H(e l ) T... the set of all possible tests A test t is optimal iff (e.g. of a given length) t = arg min t T E H(t).

6 A myopically optimal test t is a test where each question X of t minimizes the expected value of entropy after the question is answered: X = arg min X X E H(t X ), i.e. it works as if the test finished after the selected question X.

7 P ( = 0) P ( = 1) e list = {{ = 0}, { = 1}} counts[3] = P ( = 0) = 0.7 counts[1] = P ( = 1) = Myopic construction of a fixed test e list := [ ]; test := [ ]; for i := 1 to X do counts[i] := 0; for position := 1 to test lenght do new e list := [ ]; for all e e list do i := most informative X(e); counts[i] := counts[i] + P (e); for all x i X i do append(new e list, {e {X i = x i }}); e list := new e list; i := arg max i counts[i]; append(test, X i ); counts[i ] := 0; return(test);

8 CAT for basic operations with fractions Examples of tasks: T 1 : ( 3 ) = 15 1 = 5 1 = 4 = T 2 : = = 3 12 = T 3 : 1 1 = 1 3 = ) T 4 : ( ) ( = = 2 12 = 1 6.

9 Elementary and operational skills CP AD SB MT CD CL CIM Comparison (common numerator or denominator) Addition (comm. denom.) Subtract. (comm. denom.) Multiplication Common denominator Cancelling out Conv. to mixed numbers 1 > 1, 2 > = 1+2 = = 2 1 = = ( 1, ) ( = 3, ) = = = = CMI Conv. to improp. fractions = = 7 2

10 Misconceptions Label Description Occurrence MAD MSB MMT1 MMT2 MMT3 MMT4 MC a b + c d = a+c b+d 14.8% a b c d = a c b d 9.4% a b c b = a c a b c b = a+c a b c d = a d b 14.1% b b 8.1% b c 15.4% a b c d = a c b+d 8.1% a b c = a b c 4.0%

11 Student model HV1 ACL ACMI ACIM ACD CP MT CL CMI CIM CD AD SB MMT1 MMT4 MMT3 MMT2 MC MAD MSB

12 Evidence model for task T 1 ( ) 1 8 = = = 4 8 = 1 2 T 1 MT & CL & ACL & SB & MMT 3 & MMT 4 & MSB CL ACL MT SB MMT4 MSB T1 MMT3 P (X1 T1) X1

13 Skill Prediction Quality adaptive average descending ascending 88 Quality of skill predictions Number of answered questions

14 Conclusions Empirical evidence shows that educational testing can benefit from application of Bayesian networks. Adaptive tests may substantially reduce the number of questions that are necessary to be asked. Method for the design of a fixed test provided good results on tested data. It may be regarded as a good cheap alternative to computerized adaptive tests when they are not suitable. One theoretical problem related to application of Bayesian networks to educational testing is efficient inference exploiting deterministic relations in the model. This problem was topic of our UAI 2002 paper.

Bayesian Networks in Educational Testing

Bayesian Networks in Educational Testing Jiří Vomlel Institute of Information Theory and Automation Academy of Science of the Czech Republic Prague This presentation is available at: http://www.utia.cas.cz/vomlel/