emporal Knowledge Acuisition From Multiple Experts Helen Kaikova, Vagan erziyan Metaintelligence Lab., Kharkov State echnical University of Radioelectronics, 4 Lenina Avenue, 3076 Kharkov, Ukraine, e-mail: vagan@milab.kharkov.ua Abstract he paper deals with the research in the area of knowledge acuisition from multiple experts when knowledge includes temporal component. he expert s ranking refinement techniue is presented which is based on of the most supported opinion among all the experts. Paper answers to the following uestions: how to derive the most supported knowledge from the multiple experts about the unknown temporal relation between the two events; how to make uality evaluation of the most supported opinion; how to make syntactical analysis of the most supported opinion to make necessary corrections; how to make, evaluate, use and refine ranking of all the experts to improve the results.. Introduction he area of knowledge management includes the problem of eliciting expertise from more than one expert. he significance of this subject deals with fast development of telecommunications, Internet, WWW that connects people together and gives possibilities to collect knowledge from different sources. he problems, how to collect different opinions, handle inconsistent and incomplete knowledge taken from them, find consensus, support interface between individual and collective knowledge, are now under great interest of international research. Could the overlapping knowledge from multiple sources be described in such a way that it is context or even process independent? In [], the negative answer was given. If more than one expert is used, one must either select the opinion of the best expert or pool the experts judgements [, 5]. It is assumed that when experts judgements are pooled, collectively they offer sufficient cues to lead to the building of a comprehensive theory. In practice, one of the following three strategies may be used for knowledge acuisition: use the opinion of only one expert; collect the opinions of multiple experts, but use them one at a time, or integrate these opinions. Research described in [7] deals mainly with the strategy of integrating opinions. here it is assumed that acuired knowledge has more validity if it forms a consensus (if such exists) among the experts. In [6], five techniues are discussed and compared for aggregating expertise. In this study, elicited knowledge is aggregated using classical statistical methods (regression and discriminant analysis), the ID3 pattern classification method, the k-nn (Nearest Neighbour) techniue, and neural networks. In aggregating knowledge, the authors seek to identify the significance of each of the factors extracted and the functional inter-relationship among the relevant factors. A logic for reasoning with inconsistent knowledge has been described in [9]. his logic suits reasoning with knowledge coming from different and not fully reliable knowledge sources. Inconsistency may be resolved by considering the reliability of the knowledge sources used. he reliability relation can be interpreted as denoting that if two premisses are involved in a conflict the least reliable premiss has the highest probability of being wrong. he present paper deals with the integrating multiple experts opinions that include temporal relations between some events. he representation and reasoning about temporal relations are essential for knowledge representation [], natural language understanding [3,0], planning [,4], technical diagnosis [8], and other areas. An algebra with binary relations on intervals was introduced by Allen []. He gives an algorithm for computing an approximation to the strongest implied relation for each pair of intervals. We use Allen s representation of temporal relations to introduce a techniue to manage knowledge obtained from multiple knowledge sources. he scheme with problems, discussed in this paper, is shown in Figure. We consider knowledge that includes multiple experts description of some domain events by defining temporal relationships between them.
DOMAIN emporal relation Event Event Most supported opinion Experts Experts ranking Figure. he problem of knowledge acuisition from multiple experts hese problems are:. How to derive the most supported knowledge from the multiple experts about the unknown temporal relation between the two events?. How to make uality evaluation of the most supported opinion? 3. How to make syntactical analysis of the most supported opinion to make necessary corrections? 4. How to make, evaluate, use and refine ranks of all the experts to improve the results?. Basic Concepts In this chapter, we define main concepts used throughout the paper. We define knowledge about temporal domain as fifth <S,M,R,,P>. he concepts used are the following: S - the set of n knowledge sources or experts; M - the set of m basic relations for temporal points (able ). In this able, X - and X + are endpoints of the temporal interval ; Y - and Y + are endpoints of the temporal interval.; able. Set M of basic endpoints relations for two temporal intervals M Group : {,,3} M M M 3 X - < Y - X - > Y - X - = Y - Group : {4,5,6} M 4 M 5 M 6 X - < Y + X - > Y + X - = Y + Group 3: {7,8,9} M 7 M 8 M 9 X + < Y - X + > Y - X + = Y - Group 4: {0,,} M 0 M M X + < Y + X + > Y + X + = Y + R - the set of r basic relations for temporal intervals. We define this set according to Allen [,]. he set of 3 Allen s interval relations is shown in able ;
able. he set R of Allen s basic temporal relations Before R After R Meets R 3 Met-by R 4 During R 5 Includes R 6 Overlaps R 7 Overlapped by R 8 Starts R 9 Started-by Finishes R 0 R Finished by R Euals R 3 - the set of t temporal intervals which correspond to the domain events; P - the semantic predicate which defines piece of knowledge about temporal relationships in the basic domain by the following relation between the sets, R and S:, if the knowledge source Si uses Rk P ( a, Rk, b, Si) = to describe relation between a and b; 0, otherwise. 3. Deriving Most Supported Expert Opinion o derive most supported opinion among all the experts about temporal relation we first construct the RM matrix r m which defines relationship between the relations sets R and M as it is shown in able 3 by the following way: R R, M M(( R M ) RM ) k k k, ;
able 3. he RM-matrix Relationships between Allen s basic temporal intervals relations and endpoints relations RM M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M R 0 0 0 0 0 0 0 0 R 0 0 0 0 0 0 0 0 R 3 0 0 0 0 0 0 0 0 R 4 0 0 0 0 0 0 0 0 R 5 0 0 0 0 0 0 0 0 R 6 0 0 0 0 0 0 0 0 R 7 0 0 0 0 0 0 0 0 R 8 0 0 0 0 0 0 0 0 R 9 0 0 0 0 0 0 0 0 R 0 0 0 0 0 0 0 0 0 R 0 0 0 0 0 0 0 0 R 0 0 0 0 0 0 0 0 R 3 0 0 0 0 0 0 0 0 he techniue of deriving the most supported opinion concerning certain pair a and b of events is the following. he experts give their votes concerning use of each temporal relation from the set R with this pair of events. After that we make the SM a,b matrix n m which defines relationship between the set of knowledge sources S and set of basic relations M for each two fixed temporal intervals a and b by the following way:,, S S, M M, R a b i k ab, a k b i k i, R(( P(, R,, S )& ( R M )) SM ). he techniue takes into account the rank of each expert which defines the weight of this expert vote among all other votes. Let r v i will be the rank of i-th expert before v-th voting ( 0 < ri v < n ). he techniue supposes that each expert rank is initially assigned to one: ri =, i( i =,..., n). We construct the vector VOE a,b which contains results of the current experts voting concerning relation between intervals a and b derived from the matrix SM a,b as follows: ab, ab, ab, VOE = abs( ϕ ψ ), a, b, t,, m, where ab, ϕ = n ri v i, ab i( SM = ) i,, ab, ψ =, n ri v i, ab i( SM = 0) After that we derive MSUP a,b as the vector which contains most supported opinion concerning relation between intervals a and b derived as follows: ab, ab, ab, ( ϕ ψ > 0) MSUP, a, b, t,, m. he number of conflicts con v i between opinion of i-th expert and the most supported opinion is calculated through all set M during the v-th voting: coni v m = ( SMi ab,, MSUP ab, ), ab,, t, i, n, v, his number is used to refine the rank of each expert after certain vote taking into account how close are the opinion of this expert and the most supported opinion. he voting type techniue supposes that the uality of the resulting opinion is better when the number of votes that are eual to the most supported opinion is large. We make the most supported opinion uality Q evaluation by the following way: i,,.
Votes accepted as most supported opinion Q = ; All votes Q v ab, = m VOE m n i r ab, 4. Correction rules for the most supported opinion refinement Not every most supported opinion can be accepted as the result because some parts of it may include conflicts. hus we need certain syntactical analysis to handle conflicts using correction rules. We use the following correction rules to check and refine the most supported opinion:. If <all the components from the same group of the most supported opinion are eual to zero>, then <that one which has the least vote should be changed to one>. ab, ab,, t, hh, 4, (( MSUP, Group)& &( s Grouph( VOE = ab, = min( VOEGroup )))) MSUPs ab,. h. If <all the components from the same group of the most supported opinion are eual to zero and more then one of them has minimal vote in the group>, then <that one which corresponds to the relation of euivalence between temporal points should be changed to one>. ab,, t, hh, 4, (( Grouph ={,, 3} )& ab, ab, &( MSUP, Group )&((( VOE = v i. s ab, h ab, ab, ab, 3 ab, ab, ab, = VOE ) VOE ) OR(( VOE = = VOE ) VOE ))) MSUP. 3 3 3. If <there are more than one components from the same group of the most supported opinion which are eual to one>, then <that one which has the least vote should be changed to zero>.,, a, b, t, h, h 4, (( MSUP & MSUP,, s s ab ab ab, ab, ab, Grouph)& ( VOEs < VOE )) MSUPs. 4. If <there are more than one eual to one components from the same group of the most supported opinion including the last one and they have the same vote>, h then <those ones which are not correspond to the relation of euivalence between temporal points should be changed to zero>. { } a, b, t, h, h 4, (( Grouph =,, 3 )&( s ab,, Group )& ( MSUP & MSUP & h 3 s ab 3 ab, s ab, s ab, & ( VOE = VOE ))) MSUP. 3 5. Reuirements to the expert s ranking (refinement strategy) Mechanism of expert ranking is used to improve results of voting type processing of the multiple experts knowledge. he main formula used to refine ranks is the following: r + = r + r i v i v i v µ ( µ con ), where r = δ, m ( ), µ v n = v j n n con. v ri v n ri v δ i = i v i v v v his formula is based on the following strategy of refinement ranks: All the experts have the same initial rank, eual to. After each vote the rank of each expert should be recalculated. An expert improves his rank after some vote if his opinion has less conflicts with the most supported one than the average number of conflicts among all the experts. Otherwise he loses some part of his rank. An expert s rank should not be changed after some vote if expert does not participate it or his opinion has as many conflicts with the most supported one as the average number of conflicts among all the experts. An expert s rank should always be more than zero and less than number of experts. he value of punishment (prize) for presence (absence) of each conflict should be maximal for expert with rank eual to n/ (n - number of experts). he value of punishment (or prize) for presence (or absence) of each conflict should be aspire to zero for expert whose rank is close to zero or to n. 6. Example of deriving the most supported expert opinion In this chapter we will consider the example of deriving the most supported opinion of four experts concerning a temporal relation. Let the four experts are asked to express their opinion about temporal relation between two abstract temporal intervals: first vote - between intervals j i v
and ; second vote - between intervals 3 and 4 ; third vote - between intervals 5 and 6. We supposed that ranks of all experts at the very beginning (before the first vote) is eual to one. Let the experts express their opinions during the first vote by the following way: Expert : during. Expert : overlaps. Expert 3: starts. Expert 4: finished by. he results of deriving the most supported opinion for the first vote are presented in able 4. able 4. he example (first vote) SM, M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M S (R 5 ) 0 0 0 0 0 0 0 0 S (R 7 ) 0 0 0 0 0 0 0 0 S 3 (R 9 ) 0 0 0 0 0 0 0 0 S 4 (R ) 0 0 0 0 0 0 0 0 VOE 0 + + +4 +4 +4 +4 +4 +4 + +4 + MSUP 0 0 0 0 0 0 0 0 0 Correction 0 0 0 0 0 0 0 0 Results: Most supported opinion: overlaps. otal votes: 48 Positive votes: 36 Quality: 0.75 Expert ranking in the example will changed after the first vote as it is shown in able 5. able 5. Ranks refinement in the example (after first vote) Opinions M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M Expert 0 0 0 0 0 0 0 0 Expert 0 0 0 0 0 0 0 0 Expert 3 0 0 0 0 0 0 0 0 Expert 4 0 0 0 0 0 0 0 0 Most supported 0 0 0 0 0 0 0 0 Expert Number of conflicts con i Ranks refinement Rank before voting r i Rank add (lost) r i Rank after voting r i Expert - 0.065 0.9375 Expert 0 + 0.875.875 Expert 3-0.065 0.9375 Expert 4-0.065 0.9375 Average.5 0
Let the experts express their opinions during the second vote by the following way: Expert : 3 after 4 (r = 0.9375). Expert : 3 meets 4 (r =.875). Expert 3: 3 overlapped by 4 (r 3 = 0.9375). Expert 4: 3 before 4 (r 4 = 0.9375). he results of deriving the most supported opinion for the second vote are presented in able 6. able 6. he example (second vote) SM 3,4 M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M S (R ) 0 0 0 0 0 0 0 0 S (R 3 ) 0 0 0 0 0 0 0 0 S 3 (R 8 ) 0 0 0 0 0 0 0 0 S 4 (R ) 0 0 0 0 0 0 0 0 VOE +0.5 +0.5 +4 + + +4 + +0.5 +.65 +0.5 +0.5 +4 MSUP 0 0 0 0 0 0 0 0 0 Correction 0 0 0 0 0 0 0 0 Results: Most supported opinion: 3 overlaps 4. otal votes: 48 Positive votes: 0.875 Quality: 0.435 Expert ranking in the example will changed after the second vote as it is shown in able 7. able 7. Ranks refinement in the example (after second vote) Opinions M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M Expert 0 0 0 0 0 0 0 0 Expert 0 0 0 0 0 0 0 0 Expert 3 0 0 0 0 0 0 0 0 Expert 4 0 0 0 0 0 0 0 0 Most supported 0 0 0 0 0 0 0 0 Ranks refinement Expert con i δ i r i r i r 3 i Expert 6 0.957 0.9375-0.6978 0.397 Expert.33.875 + 0.487.6745 Expert 3 4 0.957 0.9375-0.396 0.7979 Expert 4 0.957 0.9375 + 0.487.356 Average 3.5 0.996 + 0.07.07
Let the experts express their opinions during the third vote by the following way: Expert : 5 includes 6 (r 3 = 0.397). Expert : 5 finished by 6 (r 3 =.6745). Expert 3: 5 after 6 (r 3 3 = 0.7979). Expert 4: 5 overlapped by 6 (r 3 4 =.356). he results of deriving the most supported opinion for the third vote are presented in able 8. able 8. he example (third vote) SM 5,6 M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M S (R 6 ) 0 0 0 0 0 0 0 0 S (R ) 0 0 0 0 0 0 0 0 S 3 (R ) 0 0 0 0 0 0 0 0 S 4 (R 8 ) 0 0 0 0 0 0 0 0 VOE +0.4 +0.4 +4.068 +3.7 +3.7 +4.068 +4.068 +4.068 +4.068 +4.068 +0.79 +0.79 MSUP 0 0 0 0 0 0 0 0 Correction 0 0 0 0 0 0 0 0 Results: Most supported opinion: 5 overlapped by 6. otal votes: 48.86 Positive votes: 3.866 Quality: 0.673 Expert ranking in the example will changed after the third vote as it is shown in able 9. able 9. Ranks refinement in the example (after third vote) Opinions M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M Expert 0 0 0 0 0 0 0 0 Expert 0 0 0 0 0 0 0 0 Expert 3 0 0 0 0 0 0 0 0 Expert 4 0 0 0 0 0 0 0 0 Most supported 0 0 0 0 0 0 0 0 Ranks refinement Expert con 3 i δ i 3 r 3 i r 3 i r 4 i Expert 0.3 0.397 0 0.397 Expert 4.98.6745-0.437.48 Expert 3 0.85 0.7979 0 0.7979 Expert 4 0.95.356 + 0.3983.7545 Average 0.9.07-0.0085.0085 One can see that after third vote the rank of first expert is such that he cannot essentially effect the most supported opinion. he rank of the fourth expert makes his opinion the most weighty among all the experts.
However it also obvious that he can lose his rank in future if all other experts will vote similarly against his opinion. 7. Example of changing the order of the domain description In this chapter we will consider the same example with same temporal intervals - 6. We will change only the order of the domain description using the same opinions of the same experts. We first consider the case of description of the temporal relation between intervals 5 and 6, then between intervals 3 and 4, and then between intervals and. he goal of such attempt is to analyse the differences in the resulting experts ranks in both examples. he results of deriving the most supported opinion and results of the experts ranks refinement for the firstthird vote in the reverse-order example are presented in ables 0-5. able 0. he reverse example (first vote) SM 5,6 M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M S (R 6 ) 0 0 0 0 0 0 0 0 S (R ) 0 0 0 0 0 0 0 0 S 3 (R ) 0 0 0 0 0 0 0 0 S 4 (R 8 ) 0 0 0 0 0 0 0 0 VOE 0 0 +4 + + +4 +4 +4 +4 +4 + + MSUP 0 0 0 0 0 0 0 0 0 Correction * * 0 0 0 0 0 0 0 Results: Most supported opinion: 5 includes 6 or 5 overlapped by 6. otal votes: 48 Positive votes: 3 Quality: 0.67 able. Ranks refinement in the reverse example (after first vote) Opinions M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M Expert 0 0 0 0 0 0 0 0 Expert 0 0 0 0 0 0 0 0 Expert 3 0 0 0 0 0 0 0 0 Expert 4 0 0 0 0 0 0 0 0 Most supported * * 0 0 0 0 0 0 0 Ranks refinement Number of Rank before voting Rank add (lost) Rank after voting Expert conflicts con i r i r i r i Expert + 0.7.7 Expert 3-0.7 0.83 Expert 3 3 + 0.7 0.83 Expert 4-0.7.7 Average 0
able. he reverse example (second vote) SM 3,4 M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M S (R ) 0 0 0 0 0 0 0 0 S (R 3 ) 0 0 0 0 0 0 0 0 S 3 (R 8 ) 0 0 0 0 0 0 0 0 S 4 (R ) 0 0 0 0 0 0 0 0 VOE 0 0 +4 +.83 +.83 +4 +.83 0 +.7 0 0 +4 MSUP 0 0 0 0 0 0 0 0 0 0 0 Correction * * 0 0 0 0 0 * * 0 Results: Most supported opinion: 3 during 4 or 3 includes 4 or 3 overlaps 4 or 3 overlapped by 4. otal votes: 48 Positive votes: 9.66 Quality: 0.4 able 3. Ranks refinement in the reverse example (after second vote) Opinions M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M Expert 0 0 0 0 0 0 0 0 Expert 0 0 0 0 0 0 0 0 Expert 3 0 0 0 0 0 0 0 0 Expert 4 0 0 0 0 0 0 0 0 Most supported * * 0 0 0 0 0 * * 0 Ranks refinement Expert con i δ i r i r i r 3 i Expert 6.037.7-0.6.009 Expert 4 0.877 0.83-0. 0.709 Expert 3 0.877 0.83 + 0.363.93 Expert 4 4.037.7-0.6.009 Average 3.5 0.9904-0. 08 0.98
able 4. he reverse example (third vote) SM, M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M S (R 5 ) 0 0 0 0 0 0 0 0 S (R 7 ) 0 0 0 0 0 0 0 0 S 3 (R 9 ) 0 0 0 0 0 0 0 0 S 4 (R ) 0 0 0 0 0 0 0 0 VOE +0.484 +.90 +.534 +3.9 +3.9 +3.9 +3.9 +3.9 +3.9 +.90 +3.9 +.90 MSUP 0 0 0 0 0 0 0 0 0 Correction 0 0 0 0 0 0 0 0 Results: Most supported opinion: overlaps. otal votes: 47.04 Positive votes: 35.65 Quality: 0.7475 able 5. Ranks refinement in the reverse example (after third vote) Opinions M M M 3 M 4 M 5 M 6 M 7 M 8 M 9 M 0 M M Expert 0 0 0 0 0 0 0 0 Expert 0 0 0 0 0 0 0 0 Expert 3 0 0 0 0 0 0 0 0 Expert 4 0 0 0 0 0 0 0 0 Most supported 0 0 0 0 0 0 0 0 Ranks refinement Expert con 3 i δ i 3 r 3 i r 3 i r 4 i Expert.006.009-0.069 0.943 Expert 0 0.778 0.709 + 0.459 0.8549 Expert 3.6.93-0.07.3 Expert 4.006.009-0.069 0.943 Average.5 0.977 0.98-0.04 0.967
One can see that the situation is changed almost totally. he first and the second votes in the reverse example have ambiguity in the selection of the most supported opinion. his happened because no one of the correction rules can be applied. One interesting fact that the second vote gave four alternatives and three of them are different from the initial opinions of experts. he resulting rank of each expert is also depends on the order of the domain description. he continuation of the experiment with the computer has been made so that new experts ranks obtained after three votes were applied with the same domain and the same order. Such iterative process of the same domain description leads after several steps to an uniue relation for each domain pair. 8. Conclusion he method can also be used to derive the most supported knowledge in domains that have a well-defined set of basic relations (such as the set M) and a set of compound relations (such as the set R). In those cases, experts are allowed to give incomplete or incorrect knowledge about a relation from the set R because it can be handled by deriving the most supported opinion of multiple experts through the set M. Ranking techniue in this research supports experts whose opinion is close to the most supported one. It even may happen that only one expert remains after several votes whose rank becomes more than all other ranks together. In many applications, the most supported knowledge is not the best one. Moreover the talent individuals can easily lose their rank if they are thinking not like others. hat is why method should be developed to pick up and classify experts whose opinions are the most different from the most supported opinion, and then try to take them into account. he uestion is also open is it good or not that the order of describing domain events effects the result of expert ranking. If one will continue the example further then he can see, that if the order of voting will changed again then ranks of experts after every three votes will be different. References. Allen, J.F., Maintaining Knowledge About emporal Intervals, Communications of the ACM, V.6, No., 983, pp. 83-843.. Allen, J.F., emporal Reasoning and Planning, In: Allen J.F. (ed.), Reasoning About Plans, Morgan Kaufmann, San Mateo, 99, pp. 8-33. 3. Allen, J.F., owards a General heory of Action and ime, Artificial Intelligence, V.3, No., 984, pp. 3-54. 4. Allen, J.F. and Koomen, J.A., Planning Using a emporal World Model, In: Proceedings of the 8 th International Joint Conference on Artificial Intelligence, Karlsruhe, 983, pp. 74-747. 5. Barrett, A.R. and Edwards, J.S., Knowledge Elicitation and Knowledge Representation in a Large Domain with Multiple Experts, Expert Systems with Applications, Pergamon, V.8, No., 995, pp. 69-76. 6. Mak, B., Bui,. and Blanning, R., Aggregating and Updating Experts Knowledge: An Experimental Evaluation of Five Classification echniues, Expert Systems with Applications, Pergamon, V.0, No., 996, pp. 33-4. 7. Medsker, L., an, M. and urban, E., Knowledge Acuisition from Multiple Experts: Problems and Issues, Expert Systems with Applications, Pergamon, V.9, No., 995, pp. 35-40. 8. Nokel, K., emporally Distributed Symptoms in echnical Diagnosis, Lecture Notes in Artificial Intelligence, V.57, Springer-Verlag, 99. 9. Roos, N., A Logic for Reasoning with Inconsistent Knowledge, Artificial Intelligence, Elsevier, V.57, No., 99, pp. 69-03. 0. Song, F. and Cohen, R., he Interpretation of emporal Relations in Narrative, In: Proceedings of the 7 th National Conference of the American Association for Artificial Intelligence, Saint Paul, 988, pp. 745-750.. aylor, W.A., Weimann, D.H. and Martin, P.J., Knowledge Acuisition and Synthesis in a Multiple Source Multiple Domain Process Context, Expert Systems with Applications, Pergamon, V.8, No., 995, pp. 95-30.. Weida, R. and Litman D., erminological Reasoning with Constraint Networks and an Application to Plan Recognition, In: Nebel B., Swartout W. and Rich C. (eds), Principles of Knowledge Representation and Reasoning: Proceedings of the 3 rd International Conference, Cambridge, 99, pp. 8-93.