A Novel Approach for Test Problem Assessment Using Course Ontology

Transcription

1 A Novel Approach for Test Problem Assessment Using Course Ontology Manas Hardas Department of Computer Science Kent State University, OH. Oct 23 rd 26 Seminar: Manas Hardas 1

2 Objective Test Design Design is useful for reengineering and reuse Problem Assessment Basic unit of test, is a problem Knowledge Content Model To answer a problem, knowledge is required Seminar: Manas Hardas 2

3 Background Web is scattered with online educational resources Mostly un-organised, but some in organised fashion as well [OCW, Universia, ACM, NSDL, CORE] Not represented in contet Looses reusability, reengineering not possible, not machine interpretable Semantic representation standards RDF ( (22) OWL ( (24) LOM ( (24) Contetual representation of problems is important Seminar: Manas Hardas 3

4 Resources represented in knowledge contet using RDF/OWL WEB Test Problems not in contet Knowledge Domains Assess Problems for knowledge Test Design Seminar: Manas Hardas 4

5 Birds eye view of the process In an Ethernet network can a second packet be transmitted as soon as the receiver receives the first packet? problem-concept mapping Objectives map concept knowledge assess test problems analyze the methodology Seminar: Manas Hardas 5

6 Scope of this talk Course Knowledge Representation Problem Assessment Results Seminar: Manas Hardas 6

7 CSG (concept space graph) Course Knowledge is represented using Concept Space Graph called as a Course Ontology Course ontology hierarchical representation of concepts taught in a course linked by has-prerequisite relationships. Each link has prerequisite, link weight Each node Self-weight, prerequisite-weight.9 Memory Management (.1) Storage Structure (.1) File System Interface (.1) OS Introduction (.3).7 File Implementation System (.1) Computer Systems Structure () OS Structures (.3).7.1 Storage Management () Virtual memory () OS Overview ().6 Distributed File Systems (.1).9 Security ().5 5 Protection and security (.15).9 Protection ().1 Distributed Coordination () O.S () Distributed Systems (.1).9 IO Systems ().5 Distributed System Structure (.1) Mass storage Structures ().6 Win XP () Case studies () Process Management () I/O System structures () Win 2 () 5 5 Linu System (.3).7 Process Synchronization () 5 5 Historical Perspective () Deadlocks () CPU Scheduling () Threads (.1).9 Epressive Computable Seminar: Manas Hardas 7

8 Course Ontology Description Language (CODL) Written in Web Ontology Language (OWL) Mostly OWL Lite with few etensions on data type properties Can represent any course ontology Basic Elements on Course Ontology OWL document are Ontology Header Class descriptions Property descriptions individuals Seminar: Manas Hardas 8

9 OS.6 Class concept relation_1 hasprerequisite hasprerequisite hasselfweight hasprerequisiteweight connectsto connectsto haslinkweight Memory Management Class Relation Linked individuals in ontology Concepts and properties in OWL Seminar: Manas Hardas 9

10 CODL individuals <Concept rdf:id="memorymanagement"/> Individuals <Concept rdf:id="os"> <hasprerequisite> <Relation rdf:id="relation_1"> <connectsto rdf:resource="#memorymanagement"/> <haslinkweight rdf:resource="#"/> </Relation> </hasprerequisite> <hasselfweight rdf:resource=".39"/> <hasprerequisiteweight rdf:resource=".61"/> </Concept> Seminar: Manas Hardas 1

11 Problem Assessment Methodology Seminar: Manas Hardas 11

12 Problem Concept Mapping [C 1, C 2,,C n ] Threshold coefficient λ input input CSG Etraction Module Projection Calculation Algorithms P(C 1 ) P(C 2 ) output P(C n ) Course ontology Problem Assessment System Concept Projections input Assessment Module Evaluation Parameters Calculation Algorithms input Assessment Parameters [α,,δ] Seminar: Manas Hardas 12

13 CSG etraction Why? CSG is very big WordNet 5, word CYC (over a million assertions) Medical/Clinical Ontology (LinKBase 1 million concepts) Selection of relevant portion of ontology to maintain computability How? Projection Graph Projection Threshold Coefficient (λ) Prunes CSG Desired semantic depth Seminar: Manas Hardas 13

14 How is routing achieved in Autonomous Systems? Projection Graph Seminar: Manas Hardas 14

15 Seminar: Manas Hardas 15 Prerequisite effect of one node over another Node Path Weight: When two concepts and t are connected through a path p consisting of nodes given by the set then the node path weight between these two nodes is given by: 1 1 1,, t m m p m m t s t W l W The node path weight for a node to itself is its self weight : 1 1 1, W s t m m,...,,...,, 1 1 Incident Path Weight: It is the the absolute prerequisite cost required to reach the root node from a subject node. Incident path weight is same as node path weight without the factor of self weight of the subject node. n n n n s n n W,,,,

16 Eample CSG(A).7.6 E B A.2.3 C F G.5 D.9 H 5 J K L M N O I.1 P.6 Seminar: Manas Hardas 16

17 Node path weight, Incident Path Weight calculations E.6 B F.7 L.3 B, L W L lf, LW F * l( B, F) W B s p * B, L.3***5*. 528 B, L/ W.528/ ( B, L) B, L W L le, LW E* l( B, E) W B s s p * B, L.3*.15*.6**. 864 B, L/ W.864/ ( B, L) s p p Seminar: Manas Hardas 17

18 Projection graph Given a root concept and a projection threshold coefficient λ, and CSG, T(C, L), a projection graph P (, λ) is defined as a sub graph of T with root and all nodes t where there is at least one path from to t in T such that node path weights satisfies the condition:, The projection set consisting of nodes concept is represented as, P for a root j Where represents the i th element of the projection set of node j. i t,..., 1 2 n,,... P, 1 2 n Seminar: Manas Hardas 18

19 Projection Calculation Eample Calculate the projection graph for Concept B, for λ=.1. B. 2.5 C J M Local K root N r L B O G P I P( B,.1) (, n) ( r, n)?.27/ Node.864 n.3192 r.86/.53 E F C J.6 E F.95 G K L M I.1 N O.3. 7 P.6. 4 Seminar: Manas Hardas 19

20 Projection Calculation P( D,.1) Local Node ( r, n) root r n D G.125 H.2125 I.5 M.357 N.475 L.28 O.34 (H).675 (I) P.135 ( r, n)? G D.9 5 H I.1 M N O P.6 Seminar: Manas Hardas 2

21 Problem Assessment Parameters Seminar: Manas Hardas 21

22 Knowledge required Coverage Coverage of a node with respect to the root node r is defined as the product of the sum of the node path weights of all nodes in the projection set P(, λ) for the concept and the self weight of and the incident projection path weight γ (r, ) from the root r. If the projection set for concept node, P(, λ) is given by then the coverage for node about the root r is defined as,,..., 1 2 n n ) ( r, ) (, m ) m ( Total coverage of multiple concepts in a problem given by set is, T C C C n C C, C..., 1 2 C n Seminar: Manas Hardas 22

23 Coverage Calculation A question connects to concepts B and D from the ontology. Find its coverage. B A.2 C ( B) ( B) ( B) ( A, B)* i ( B, P B i (*.98)*( ) ).7 J.6 E F.95 G K L M I.1 N O.3. 7 P.6. 4 Seminar: Manas Hardas 23

24 .98 G.5 A D.9 5 H I.1 M N O P.6 ( D) ( A, D)* ( D, P ( D) (*.98)*(.56625) ( D) ( total) i BD D i ( total) ( total) ) Seminar: Manas Hardas 24

25 Diversity The breadth of knowledge domain Opposite of similarity The ratio of summation of node path weights of all nodes in the non-overlapping set to their respective roots, and the sum of the summation of node path weights of all nodes in the overlap set and summation of node path weights of all nodes in the non-overlap set. Diversity, Where, Concept set, 1 C C C C 1 C1 C1 Projection sets, P( C, ) [,,..., ], 1 2 P( C, ) [,,..., ] a b C n C n C n P C, ) [,,..., ] Overlap set, Non-overlap set, C C C, C... O O O, O... q m1, 1 2, 2 O j q N N N, N... p m1 p j i j, Om i, N m, 1 2 i, N i m m1 C n N i p where i, j C ( n 1 2 c Seminar: Manas Hardas 25

26 Diversity Calculation.7.6 E B. 2 C F G.5 D.9 H 5 J K L M N O I.1 P.6 N [ B, C, D, E, F, G, H, J, K, L] O [ I, M, N, O, P] ( n, O ( n, N e ) e ) ( n, N e ) n B D; e element of set Seminar: Manas Hardas 26

27 Conceptual Distance Measures similarity between concepts i.e. distance from ontology root. It is defined as the log of inverse of the minimum value of incident path weight (maimum value of threshold coefficient) which is required to encompass all the concepts from the root concept. If question asks concept set the root concept r is given by, C, C... C 1 C C C, C... n log, min C n then the conceptual distance from 1 r, C, r, C... r, C 1 n Greater the distance between the concepts, more is the semantic depth. Seminar: Manas Hardas 27

28 Conceptual Distance Calculation Calculate conceptual distance between (E, F, M).98 A.2.3 ( E, F, M ) log2 min 1 A, E, A, F, A, M.6 E B. 2.5 F.95 C G.5 5 D ( E, F, M ) log2 min.1568,156, ( E, F, M ) log ( E, F, M ) M.3 Seminar: Manas Hardas 28

29 Results and Parameter Performance Analysis Seminar: Manas Hardas 29

30 Setting Operating system course ontology created using prescribed tet books OSOnto (>135 concepts) XML and OWL 4 quizzes, 38 questions composed using concepts selected from OS Ontology Tests administered by at least 25 graduate and undergraduate students Scoring done by at least 2 graders per question and average score taken. Do the parameters provide any insight into the perceived difficulty/compleity of the question? Performance analysis = Plotting average score/parameter values Seminar: Manas Hardas 3

31 coverage vs. average score coverage normalized avg. score 5 3 threshold coeff.λ=(.2) λ= (5E-3) λ= (5E-5) λ= (5E-7) λ= (5E-9) normalized avg. score question -1 coverage and average score inversely correlated behavior constant for changing threshold coefficient Seminar: Manas Hardas 31

32 Diversity vs. average score diversity normalized avg. score λ=(.2) λ=(5e-3) λ=(5e-5) λ=(5e-7) λ=(5e-9) normalized avg. score question -15 diversity and average score inversely correlated behavior constant for changing threshold coefficient Seminar: Manas Hardas 32

33 Conceptual distance vs. average score conceptual distance 11 normalized avg. score 9 distance normalized avg. score question -1 conceptual distance and average score inversely correlated distance does not vary with threshold coefficient Seminar: Manas Hardas 33

34 Correlation study.3 coverage-average score correlation threshold coefficient diversity-average score correlation distance-average score correlation threshold coefficient coverage-avg. score correlation decreases with threshold coefficient diversity-avg. score correlation decreases with threshold coefficient Seminar: Manas Hardas 34

35 Observations and Inferences (Coverage, diversity and conceptual distance) α (1/Average score) Indicates perceived difficulty Coverage gives the knowledge required Diversity indicates the scope and the breadth of knowledge domain Distance gives the relationship of the concepts with the ontology root and a pseudo similarity measure Threshold coefficient plays important role Coverage and diversity values change according to threshold coefficient Threshold coefficient changes the projection graph to desired semantic significance Conceptual Distance behavior is same for changing threshold coefficient values as it is independent of the projection graph. Gives an inverse similarity measure for subject concepts with respect to ontology root (rather than local root, for which definition can be easily etended). Seminar: Manas Hardas 35

36 Qualitative Data Analysis Questions are sorted according to those with high inverse correlation and those with lower inverse correlation between coverage-average score λ=(.2) λ=(5e-3).2 9 λ=(5e-5) coverage normalized avg. score λ=(5e-7) λ=(5e-9) normalized avg. score questions -1 coverage normalized avg. score λ=(.2) λ=(5e-3) λ=(5e-5) λ=(5e-7) λ=(5e-9) avg. score question Seminar: Manas Hardas 36

37 Questions sorted according to diversity Diversity Problems average score diversity(.2) diversity(5e-3) diversity(5e-5) diversity(5e-7) diversity(5e-9) normalized avg. score diversity average score diversity(.2) diversity(5e-3) diversity(5e-5) diversity(5e-7) diversity(5e-9) normalized avg. score problem -1 Seminar: Manas Hardas 37

38 Correlation based analysis problems B concepts high problem-concept inverse corelation low problem-concept inverse correlation Region 2 Region 1 A Large clustering (big circle) Dispersed concepts distribution and Diversity. Small Clustering Quiz based concepts distribution (2-4 and 75-1) more Seminar: Manas Hardas 38

39 Test based analysis concepts Problems most problems contain concepts in and around 2-4 and 7-1 concepts in problems go on increasing clustering denote projections of mapped concepts Seminar: Manas Hardas 39

40 Conclusions: For an automatic test design system and assessment framework is a must. To make course ware resources reusable and machine interpretable they have to represented in contet. Semantic representation standards like RDF and OWL are used to represent this contet. A representation language schema for course knowledge representation using ontology is given. The language is in OWL Lite and is epressible and computable. Problem compleity and knowledge content can be computed by applying synthetic parameters to course ontology having known the concept mapping. It is observed that the parameters are pretty good indicators of problem compleity. Assessment system can be intuitively be applied to automatic test design. Seminar: Manas Hardas 4

41 Related Work Problem assessment Li, Sambasivam Static knowledge structure Rita Kuo et. al. Information objects of simple questions Cognitive Lee, Heyworth Difficulty factors (perceived steps, students degree of familiarity, operations and epression in a problem) Koedinger, Heffernan et.al. number of symbols, ambiguous language Seminar: Manas Hardas 41

42 References: Javed I. Khan, Manas Hardas, Yongbin Ma, "A Study of Problem Difficulty Evaluation for Semantic Network Ontology Based Intelligent Courseware Sharing" WI, pp , the 25 IEEE/WIC/ACM International Conference on Web Intelligence (WI'5), 25. Jaakkola, T., Nihamo, L., Digital Learning Materials do not Possess knowledge: Making critical distinction between information and knowledge when designing digital learning materials for education, International Standards Organization, Versailles, 23. Lee, F.-L, Heyworth, R., Problem compleity: A measure of problem difficulty in algebra by using computer. Education Journal Vol 28, No.1, 2. Li, T and S E Sambasivam. Question Difficulty Assessment in Intelligent Tutor Systems for Computer Architecture. In The Proceedings of ISECON 23, v 2 (San Diego): ISSN: (Also appears in Information Systems Education Journal 1: (51). ISSN: X.) Kuo, R., Lien, W.-P., Chang, M., Heh, J.-S., Analyzing problem difficulty based on neural networks and knowledge maps. International Conference on Advanced Learning Technologies, 24, Education Technology and Society, 7(2), Rita Kuo, Wei-Peng Lien, Maiga Chang, Jia-Sheng Heh, Difficulty Analysis for Learners in Problem Solving Process Based on the Knowledge Map. International Conference on Advanced Learning Technologies, 23, MIT OCW Program Evaluation Findings Report (March 25). OWL Web Ontology Language Reference, Mike Dean and Guus Schreiber, Editors, W3C Recommendation, 1 February 24, /. Latest version available at Seminar: Manas Hardas 42

43 Thank you. Questions??? Seminar: Manas Hardas 43