Bayesian Networks in Educational Testing

Size: px

Start display at page:

Download "Bayesian Networks in Educational Testing"

Blanche Parks
5 years ago
Views:

1 Bayesian Networks in Educational Testing Jiří Vomlel Laboratory for Intelligent Systems Prague University of Economics This presentation is available at:

2 Contents: Educational testing is a big business. What is a fixed test and an adaptive test? An example: a test of basic operations with fractions. Optimal and myopically optimal tests. Construction of a myopically optimal fixed test. Results of experiments. Ane example showing that modeling dependence between skills is important. Conclusions.

3 Educational Testing Service (ETS) Educational Testing Service is the world s largest private educational testing organization with 2,300 regular employees. Volumes for ETS s Largest Exams in : 3,185,000 SAT I Reasoning Test and SAT II: Subject Area Tests (the SAT test is the standard college admission test in US) 2,293,000 PSAT: Preliminary SAT/National Merit Scholarship Qualifying Test 1,421,000 AP: Advanced Placement Program 801,000 The Praxis Series: Professional Assessments for Beginning Teachers and Pre-Professional Skills Tests 787,000 TOEFL: Test of English as a Foreign Language 449,000 GRE: Graduate Record Examinations General Test etc.

4 Fixed Test vs. Adaptive Test Q 1 Q 2 Q 5 Q 3 wrong correct Q 4 wrong Q 4 correct wrong Q 8 correct Q 5 Q 2 Q 7 Q 6 Q 9 Q 6 wrong correct wrong correct wrong correct wrong correct Q 7 Q 1 Q 3 Q 6 Q 8 Q 4 Q 7 Q 7 Q 10 Q 8 Q 9 Q 10

5 Computerized Adaptive Testing (CAT) Objective: Two basic steps: An optimal test for each examinee (1) examinee s knowledge level is estimated (2) questions appropriate for the level are selected. R. Almond and R. Mislevy from ETS proposed to use graphical models in CAT. one student model (relations between skills, abilities, etc.) several evidence models, one for each task or question.

6 CAT for basic operations with fractions T 1 : ( ) Examples of tasks: 1 8 = = = 4 8 = 1 2 T 2 : = = 3 12 = T 3 : = 4 32 = 3 8 ( ) ( ) T 4 : = 4 23 = 12 2 = 6 1.

7 Elementary and operational skills CP Comparison (common numerator or denominator) 1 2 > 1 3, 2 3 > 1 3 AD Addition (comm. denom.) = = 3 7 SB Subtract. (comm. denom.) = = 1 5 MT Multiplication = 3 CD Common denominator ( 12, 2 3 ) CL Cancelling out 4 6 = = = ( 36, 4 6 ) CIM Conv. to mixed numbers 7 2 = = CMI Conv. to improp. fractions = = 7 2

8 Misconceptions Label Description Occurrence MAD MSB MMT1 MMT2 MMT3 MMT4 MC a b + d c = b+d a+c 14.8% a b d c = b d a c 9.4% a b a b a b a b cb = a c cb = a+c cd = a d cd = a c b 14.1% b b 8.1% b c 15.4% b+d 8.1% a b c = a b c 4.0%

9 Process that lead to the student model decision on what skills will be tested, preparation of paper tests paper tests given to students at Brønderslev high school, 149 students did the test. analysis of results, finding misconceptions, summarizing results into a data file, learning a Bayesian network model using the PC-algorithm and the EM-algorithm, attempts to explain some relations between skills and misconceptions using hidden variables, a new learning phase with hidden variables included, certain edges required to be part of the learned model.

10 Student model HV2 HV1 ACMI ACIM ACL ACD AD SB CMI CIM CL CD MT CP MAD MSB MC MMT1 MMT2 MMT3 MMT4

11 Evidence model for task T 1 ( ) 1 8 = = = 4 8 = 1 2 T 1 MT & CL & ACL & SB & MMT3 & MMT4 & MSB CL ACL MT SB MMT3 MSB T 1 MMT4 P( T 1 )

12 Student + Evidence model HV2 HV1 ACMI ACIM ACL ACD AD SB CMI CIM CL CD MT CP MAD MSB MC MMT1 MMT2 MMT3 MMT4 T 1

13 Example of an adaptive test X 3 = yes X 2 = yes X 2 : 1 5 < 1 4? X 2 = no X 3 : 1 4 < 2 5? X 3 = no = yes : 1 5 < 2 5? = no Entropy of a probability distribution P(S i ) H (P(S i )) = s i S i P(S i = s i ) log P(S i = s i ) Total entropy in a node n: H(e n ) = Si S H(P(S i e n )). Expected entropy at the end of a test t is EH(t) = l L(t) P(e l ) H(e l ).

14 X 2 X 3 Let T be the set of all possible tests. A test t is optimal iff X 2 X 3 A selected test t = arg min t T EH(t). X 2 X 3 X 3 A myopically optimal test t is a test where each question X of t minimizes the expected value of entropy after the question is answered: X 3 X 2 X = arg min X X EH(t X), X 2 i.e. it works as if the test finished after the selected question X.

15 X 2 X 3 X 2 X 3 P(X 2 = 0) X 3 X 2 X 3 P(X 2 = 1) X 2 X 3 X 2 e list = {{X 2 = 0}, {X 2 = 1}} counts[3] = P(X 2 = 0) = 0.7 counts[1] = P(X 2 = 1) = 0.3 X 2 X 3... Myopic construction of a fixed test e list := [ ]; test := [ ]; for i := 1 to X do counts[i] := 0; for position := 1 to test lenght do new e list := [ ]; for all e e list do i := most in f ormative X(e); counts[i] := counts[i] + P(e); for all x i X i do append(new e list, {e {X i = x i }}); e list := new e list; i := arg max i counts[i]; append(test, X i ); counts[i ] := 0; return(test);

16 Skill Prediction Quality adaptive average descending ascending 88 Quality of skill predictions Number of answered questions

17 Total entropy of probability of skills adaptive average descending ascending 10 Entropy on skills Number of answered questions

18 Question Prediction Quality adaptive average descending ascending Quality of question predictions Number of answered questions

19 An example of a simple diagnostic task Diagnosis of the absence or the presence of three skills S 1, S 2, S 3 by use of a bank of three questions,2,,3, X 2,3. such that P(X i, j = 1 S i = s i, S j = s j ) = 1 if (s i, s j ) = (1, 1) 0 otherwise. Assume answers to all questions from the item bank are wrong, i.e.,2 = 0,,3 = 0, X 2,3 = 0.

20 Reasoning assuming skill independency,3 All skills are independent S 1 S 3 P(S 1, S 2, S 3 ) = P(S 1 ) P(S 2 ) P(S 3 ),2 S 2 X 2,3 and P(S i ), i = 1, 2, 3 are uniform. Then the probabilities for j = 1, 2, 3 are: P(S j = 0,2 = 0,,3 = 0, X 2,3 = 0) = 0.75, i.e. we can not decide which skills are present and which are missing.

21 Modeling dependence between skills,3 S 1 S 3 with deterministic hierarchy,2 S 2 X 2,3 S 1 S 2, S 2 S 3 P(S 1 = 0,2 = 0,,3 = 0, X 2,3 = 0) = 1 P(S 2 = 0,2 = 0,,3 = 0, X 2,3 = 0) = 1 P(S 3 = 0,2 = 0,,3 = 0, X 2,3 = 0) = 0.5 Observe, that for i = 1, 2, 3 P(S i,2 = 0,,3 = 0, X 2,3 = 0) = P(S i X 2,3 = 0), i.e. X 2,3 = 0 gives the same information as,2 = 0,,3 = 0, X 2,3 = 0.

22 Conclusions Empirical evidence shows that educational testing can benefit from application of Bayesian networks. Adaptive tests may substantially reduce the number of questions that are necessary to be asked. The new method for the design of a fixed test provided good results on tested data. It may be regarded as a good cheap alternative to computerized adaptive tests when they are not suitable. One theoretical problem related to application of Bayesian networks to educational testing is efficient inference exploiting deterministic relations in the model. This problem was addressed in our UAI 2002 paper.

23 ... and this is the END. It s time to have a beer.... or are there any questions?

Bayesian Networks in Educational Testing

Bayesian Networks in Educational Testing Jiří Vomlel Institute of Information Theory and Automation Academy of Science of the Czech Republic This presentation is available at: http://www.utia.cas.cz/vomlel/slides/pgm02slides.pdf