Bayesian Networks in Educational Testing

Size: px

Start display at page:

Download "Bayesian Networks in Educational Testing"

Dylan Burns
5 years ago
Views:

1 Bayesian Networks in Educational Testing Jiří Vomlel Institute of Information Theory and Automation Academy of Science of the Czech Republic Prague This presentation is available at:

2 Contents: What is a Bayesian network? Student model and evidence models. A case study: Students perfoming computations with fractions. Variables in the student model. Construction of the student model. Evidence models. Construction of an adaptive test. Construction of a fixed test. Results of experiments.

3 Bayesian network X 1 P (X 1 ) X 2 P (X 2 ) P ( X 1 ) X 4 P (X 4 X 2 ) X 5 P (X 5 X 1 ) X 6 P (X 6, X 4 ) X 7 X 8 X 9 P (X 7 X 5 ) P (X 8 X 7, X 6 ) P (X 9 X 6 ) P (X 1,..., X 9 ) = = P (X 9 X 8,..., X 1 ) P (X 8 X 7,..., X 1 )... P (X 2 X 1 ) P (X 1 ) = P (X 9 X 6 ) P (X 8 X 7, X 6 ) P (X 7 X 5 ) P (X 6 X 4, ) P (X 5 X 1 ) P (X 4 X 2 ) P ( X 1 ) P (X 2 ) P (X 1 )

4 Student and evidence models (R. Almond and R. Mislevy, 1999) S 1 S 2 S 3 S 4 S 5 S 1 S 3 S 3 S 4 S 2 S 5 X 1 X 2

5 Sudents solving problems with fractions A group of university students from Aalborg University prepared paper tests that were given to students at Brønderslev High School. Four elementary skills, four operational skills, and abilities to apply operational skills to complex tasks were tested. 149 students solved the test. The tests were analyzed in detail so that it was possible for most of students to decide whether they have or have not the tested skills. Several different models were learned using the PC algorithm Models were compared on how well they predict skills using the leave-one-out method.

6 Examples of tasks - operations with fractions T 1 : ( 3 ) = 15 1 = 5 1 = 4 = T 2 : = = 3 12 = T 3 : 1 1 = 1 3 = ) T 4 : ( ) ( = = 2 12 = 1 6.

7 Elementary and operational skills CP AD SB MT CD CL CIM Comparison (common numerator or denominator) Addition (comm. denom.) Subtract. (comm. denom.) Multiplication Common denominator Cancelling out Conv. to mixed numbers 1 > 1, 2 > = = = 2 1 = = ` 1 2, 2 3 = ` 3 6, = 2 2 = = = CMI Conv. to improp. fractions = = 7 2

8 Misconceptions Label Description Occurrence MAD MSB MMT1 MMT2 MMT3 MMT4 MC a b + c d = a+c b+d 14.8% a b c d = a c b d 9.4% a b c b = a c a b c b = a+c a b c d = a d b 14.1% b b 8.1% b c 15.4% a b c d = a c b+d 8.1% a b c = a b c 4.0%

9 Student model HV1 ACL ACMI ACIM ACD CP MT CL CMI CIM CD AD SB MMT1 MMT4 MMT3 MMT2 MC MAD MSB

10 Evidence model for task T «1 8 = = = 4 8 = 1 2 T 1 MT & CL & ACL & SB & MMT 3 & MMT 4 & MSB CL ACL MT SB MMT4 MSB T1 MMT3 P (X1 T1) X1

11 The overal model

12 See the model in Hugin:

13 Tested models (a) (b) (c) (d) (e) (f) (g) (h) two hidden variables with several restrictions on presence or absence of edges, two hidden variables and only few restrictions, one hidden variable and few restrictions, no hidden variable and few restrictions, no hidden variable and only obvious logical constraints as restrictions, no hidden variable and independent skills, Naïve Bayes model with one hidden variable (with two states) being parent of all skills, as above, but the hidden variable has three states,

14 Comparison of different models using t values vs. (b) vs. (c) vs. (d) vs. (e) vs. (f) vs. (g) vs. (h) total (a) (b) (c) (d) (e) (f) (g) (h) +3

15 Fixed Test vs. Adaptive Test Q 1 Q 2 Q 5 Q 3 wrong correct Q 4 wrong Q 4 correct wrong Q 8 correct Q 5 Q 6 wrong Q 2 correct wrong Q 7 correct wrong Q 6 correct wrong Q 9 correct Q 7 Q 1 Q 3 Q 6 Q 8 Q 4 Q 7 Q 10 Q 7 Q 8 Q 9 Q 10

16 Entropy as an information criteria 0.7 -x*log(x)-(1-x)*log(1-x) Entropy of probability distribution P (S) on skills S is defined as H (P (S)) = s P (S = s) log P (S = s) The lower the entropy the more we know about a student.

17 X 1 X 2 X 2 X 2 X 1 X 1 X 1 X 2 X 1 X 2 Entropy in node n H(e n ) = H(P (S e n )) Expected entropy at the end of test t E H (t) = l L(t) P (e l ) H(e l ) T... the set of all possible tests A test t is optimal iff (e.g. of a given length) t = arg min t T E H(t).

18 A myopically optimal test t is a test where each question X of t minimizes the expected value of entropy after the question is answered: X = arg min X X E H(t X ), i.e. it works as if the test finished after the selected question X.

19 X 2 X 1 X 2 P (X 2 = 0) X 1 X 2 X 1 P (X 2 = 1) X 1 X 2 X 1 X 2 e list = {{X 2 = 0}, {X 2 = 1}} counts[3] = P (X 2 = 0) = 0.7 counts[1] = P (X 2 = 1) = 0.3 X 2... Myopic construction of a fixed test e list := [ ]; test := [ ]; for i := 1 to X do counts[i] := 0; for position := 1 to test lenght do new e list := [ ]; for all e e list do i := most informative X(e); counts[i] := counts[i] + P (e); for all x i X i do append(new e list, {e {X i = x i }}); e list := new e list; i := arg max i counts[i]; append(test, X i ); counts[i ] := 0; return(test);

20 Entropy of the probability distributions on the skills adaptive average descending ascending 10 Entropy on skills Number of answered questions

21 Skill Prediction Quality adaptive average descending ascending 88 Quality of skill predictions Number of answered questions

22 Conclusions Empirical evidence shows that educational testing can benefit from application of Bayesian networks. Adaptive tests may substantially reduce the number of questions that are necessary to be asked. Method for the design of a fixed test provided good results on tested data. It may be regarded as a good cheap alternative to computerized adaptive tests when they are not suitable. One theoretical problem related to application of Bayesian networks to educational testing is efficient inference exploiting deterministic relations in the model. This problem is a topic of our current research.

Bayesian Networks in Educational Testing

Bayesian Networks in Educational Testing Jiří Vomlel Institute of Information Theory and Automation Academy of Science of the Czech Republic This presentation is available at: http://www.utia.cas.cz/vomlel/slides/pgm02slides.pdf