Information Evaluation in fusion: a case study. Laurence Cholvy. ONERA Centre de Toulouse, BP 4025, 2 av Ed Belin, Toulouse, France

Size: px

Start display at page:

Download "Information Evaluation in fusion: a case study. Laurence Cholvy. ONERA Centre de Toulouse, BP 4025, 2 av Ed Belin, Toulouse, France"

Stuart Davis
6 years ago
Views:

1 Information Evaluation in fusion: a case study Laurence Cholvy ONERA Centre de Toulouse, BP 4025, 2 av Ed Belin, Toulouse, France cholvy@cert.fr Abstract. This paper examines a case study of information fusion 1.. It focuses on some recommendations dened by NATO relatively to information evaluation in intelligence. These recommendations, written in natural language, are ambiguous and imprecise and we discuss the fact that they dene a scale for evaluating information. We then propose a model of information evaluation, which takes into account the main notions which underline the recommendations and we show its use in the fusion process. Keywords: application, information fusion 1 Introduction An important step in the intelligence gathering process is the fusion of information provided by several sources. The objective of this process is to build an up-to-date and correct view of the current situation with the overall available information in order to make adequate decisions. Moreover, to succeed in this process, it is important to associate with each available information, some attributes like the number of the sources that support it, their reliability, the degree of truth of the information etc. For the moment, these attributes are managed by the human operator when he fuses information provided by the dierent sources. But there is no real methodology to do this in a formal manner. And, when relaying this fusion process to a machine, we need to develop formal denitions and algorithms to manage these attributes in addition to fusing information. Furthermore, in a context of interoperability where dierent systems exchange information, common denitions of these attributes have to be shared. The Standardization Agreements (STANAG) 2022 of North Atlantic Treaty Organization (NATO) denes a framework for such common denitions. The aim of this paper is to analyse the STANAG 2022 recommendations about information evaluation and to dene a formal and non ambiguous system for evaluating information. Indeed, as it will be shown, the present recommendations, written in natural language are rather ambiguous and imprecise and are subject to discussion. 1 This work was supported by SPOTI (Service des Programmes d'observation de Telecommunication et d'information) grant DTIM/PTF/001/01.02

2 Consequently, the main contribution of this paper is not a new theory for information fusion, but is to show how to adapt some existing theory to the case of information evaluation in intelligence agreeing with recommendations of NATO. This paper is organized as follows. Section 2 presents STANAG 2022 recommendations. It also lists some criticisms that can be done and points out the main notions that underline these recommendations. Section 3 presents a formal model for evaluating information which tends to agree with these notions. Section 4 shows the application of this model in information fusion process. An example is described in section 5. Finally, section 6 is devoted to a discussion. 2 Discussion on STANAG 2022 recommendations The Annex to STANAG 2022 (Edition 8) [1] explicitly mentions that the aim of information evaluation is to indicate the degree of condence that may be placed in any item of information which has been obtained for intelligence. (...) This is achieved by adopting an alphanumeric system of rating which combines a measurement of the reliability of the source of information with a measurement of the credibility of that information when examined in the light of existing knowledge. Examining the whole text leads us to point out that the two main notions of this evaluation system are the reliability of the sources and the credibility of the information. These notions are dened in the STANAG 2022 recommendations, as follows: Reliability of the source is designated by a letter between A and F signifying various degrees of condence as indicated below. { a source is evaluated A if it is completely reliable. It refers to a tried and trusted source which can be depended upon with condence { a source is evaluated B if it is usually reliable. It refers to a source which has been successfully used in the past but for which there is still some element of doubt in particular cases. { a source is evaluated C if it is fairly reliable. It refers to a source which has occasionally been used in the past and upon which some degree of condence can be based. { a source is evaluated D if it is not usually reliable. It refers to a source which has been used in the past but has proved more often than not unreliable. { a source is evaluated E if it is unreliable. It refers to a source which has been used in the past and has proved unworthy of any condence. { a source is evaluated F if its reliability cannot be judged. It refers to a source which has not been used in the past. Credibility of information is designated by a number between 1 and 6 signifying varying degrees of condence as indicated below. { If it can be stated with certainty that the reported information originates from another source than the already existing information on the same subject, then it is classied as \conrmed by other sources" and rated 1.

3 { If the independence of the source of any item of information cannot be guaranteed, but if, from the quantity and quality of previous reports, its likelihood is nevertheless regarded as suciently established, then the information should be classied as \probably true" and given a rating of 2. { If, despite there being insucient conmation to establish any higher degree of likelihood, a freshyly reported item of information does not conict with the previously reported behaviour pattern of the target, the item may be classied as \possibly true" and given a rating of 3. { An item of information which tends to conict with the previously reported or established behaviour pattern of an intelligence target should be classied as \doubtful" and given a rating of 4 { An item of information which positively contradicts previously reported information or conicts with the established behaviour pattern of an intelligence target in a marked degree should be classied as \improbable" and given a rating of 5 { An item of information is given a rating of 6 if its truth cannot be judged. The previous denitions of information evaluation can be discussed. Indeed, since they are given in natural language, they are quite imprecise and ambiguous. Let us discuss three points. According to the previous recommendations, the reliability of a source is dened in reference to its use in the past. It can be measured for example, as the ratio of the number of times it gave a true information to the number of times it gave information. However, this denition does not take into account the actual environment of use of the information source. For instance, even if it is known to be reliable, an infra-red sensor loose reliability when it rains. Concerning the credibility of information, it seems that the rating dened according to the recommendations does not qualify an unique property. For instance, how should we note an item of information supported by several sources of information which is also conictual with some already registered information? According to these denitions, this item should be given a rating of 1 and should also be given a rating of 5. Furthermore, according to the recommendations, a rating of 6 should be given to an item whose truth cannot be judged. This suppposes that the other ratings (1...5) concern the evaluation of the truth of information. If so, rating of 1 should be given to a true information. But, as it is dened, rating of 1 should be given to an item supported by at least two sources. This is questionable since, the dierent sources (even if they agree) may emit false information. However, even if the previous recommendations can be discussed, one can see that the three main notions which underline the evaluation system are : { the number of sources that support an information { their reliability { the fact that the information tends to conict or conicts with some available information

4 These three characteristics are independent. For instance, two sources, one being fully reliable and the other not usually reliable may have emitted the same piece of information. Furthermore, this piece of information may be contradictory with another piece of information. In the following, we present a model of evaluation which takes into account the number of the sources and their reliability but which does not make a distinction between \tending to conict" and \conicting". In this model, that unique notion of conict is represented by the property of inconsistency. 3 Proposal for a formal model of evaluation In order to dene an evaluation as close as possible to the one recommended by the STANAG, an information evaluation will still be a pair. But the meaning of the two members of that pair will be dierent. This meaning is given below. We consider a propositional language L, used to express information. Information will then be formulas of L. M od(f ) will demote the set of models of the formula F. We consider several sources of information which deliver information about an observed situation. Each source of information is associated with its degree of reliability, which is a letter belonging to fa; :::; Eg depending on the fact that this source is judged more or less reliable. Notice that this denition could be rened by dening the degree of reliability of a source according to the probability that this source gives true information in the current context of use. Rening this denition is not necessary for the following. Each information emitted by a source is stored in a database DB with its evaluation as dened by the following denition. Denition. The evaluation of a formula F, denoted eval(f ), is a pair < V; m >, such that : { V is a vector of letters belonging to fa; B; C; D; Eg. The length of this vector indicates the number of sources which support the information and the contents of the vector reects their reliability degrees. { m is an integer which is equal to 1 i the information is consistent with the other avalaible information stored in DB or equal to 2 else. Examples. The evaluation < (A); 1 > is associated with an information which has been emitted by a completely reliable source and which is consistent with the current state of DB. The evaluation < (A; C); 2 > is associated with an information which is supported by two sources, one being completely reliable, the other one being fairly reliable, and which is inconsistent with the current state of DB. One can notice that in a given state of DB, the evaluation of an information is unique. However, an information evaluation may change during time as it will be shown in the following.

5 4 Application of this model in fusion In this section, we show how to apply this previous model is the process of information fusion. That process is the following: as soon as an item of information is emitted by a source (sensor, human), it is stored in DB with its appropriate evaluation. From time to time, queries are asked to DB in order to make decisions. Besides DB, we also have a set of constraints, IC, stating what is true for certain in the real world. For instance IC may contain the following rules if the observed object is a plane, then it is not an helicopter, if a vehicule moves north, then it does not move south, etc Registering an information into DB Let us consider an information F emitted by a source whose reliability is r. The following algorithm describes the registration, in DB, of F associated with its evaluation. { If F is not yet stored in DB. F is consistent with DB [ IC. Then F is stored in DB with the evaluation << r >; 1 >. F is inconsistent with DB [ IC. Then F is stored in DB with the evaluation << r >; 2 >. Let S i = ffi 1 ni ; :::Fi g (i = 1...n) the minimal subsets of DB inconsistent with F [ IC. Then 8i = 1:::n 8j = 1:::n i, if the evaluation of Fi j was < V j i ; m > then it is updated and becomes < Vi j ; 2 >. { If F is already stored in DB. Assume that F is stored in DB with the evaluation << V >; m >. Then this evaluation becomes < V 0 ; m > where V 0 is the vector obtained by agregating V with r. Notice that the length of the vector has been incremented by 1 (indeed, one more source supports F ), but m is not changed : even if one more source emitted F, this does not change the fact that F is (or not) consistent with DB [ IC. Remark. In this last case, we should also check the independance of the dierent sources of information which have emitted F. But, in order to simplify the presentation of the model, we assume independence of these sources. So, nally DB is a nite set of formulas associated with their evaluation. We can then write DB = f< F 1 ; eval(f 1 ) > :::: < F n ; eval(f n ) >g. 4.2 Querying DB The problem is now to answer a query addressed to DB. Let Q be a formula expressing this query. We denote jjqjj DB;IC the answer to Q addressed to DB under the constraints IC.

6 In the following, we show that given IC, DB can be associated with an unique consistent set of formulas of L, denoted merge(db; IC) and we will dene jjqjj DB;IC according to the satisfaction of Q in this set. merge(db; IC) is here dened by extending a majority merging operator by taking into account weigthed sum instead of sum. The weigths here will be numbers (5, 4, 3, 2, 1) which intend to represent, on a numerical scale, the dierent degrees of reliability of the sources (A, B, C, D, E). Denition of W We recall here some denitions introduced by Konieczny and Pino-Perez [2] [3] and adapt them for our purpose. Let db 1...db n be n sets of formulas of L and IC a set of integrity constraints. Konieczny and Pino-Perez dened a majority merging operator, denoted ;IC, such that the set of models of the information source which is obtained from merging db 1... db n with this operator, is semantically characterized by: M od( ;IC ([db 1 ; :::; db n ])) = min [db 1 :::dbn ] (Mod(IC)) [db 1:::db n] is a total pre-order on M od(ic) dened by: w [db 1:::db n] w 0 i d (w; [db 1 :::db n ]) d (w 0 ; [db 1 :::db n ]) with d (w; [db 1 :::db n ]) = nx i=1 min d(w; w 0 ) w 0 2Mod(db i) where d(w; w 0 ) is the Hamming distance (i.e, the number of propositional letters whose valuation diers from w to w 0 ). In other words, when merging db 1 :::db n with the operator ;IC, the result is semantically characterized by the models of IC which are minimal according to the pre-order [db 1;:::;db n]. As Konieczny and Pino-Perez explicitely mentionned it in conclusion of [2], this merging operator ;IC can be extended by taking into account a weigthed sum instead a simple sum However, the properties of this operator remain to be studied. In the following, we suggest such an extension, which fully agrees with the weighted model-tting approach of Revesz [4] to the revision of a (possibly inconsistent) weighted knowldege-base by a formula. We assume now that any db i is associated with an integer denoted r(db i ). We dene W;IC by : M od( W;IC ([db 1 ; :::; db n ])) = min W [db 1 :::dbn ] (M od(ic)) W [db 1:::db n] is a total pre-order on M od(ic) dened by:

7 with i.e, w W [db 1:::db n] w 0 i d W (w; [db 1 :::db n ]) d W (w 0 ; [db 1 :::db n ]) d W (w; [db 1 :::db n ]) = d W (w; [db 1 :::db n ]) = nx i=1 nx i=1 min d(w; w 0 ):r(db i ) w 0 2Mod(db i) r(db i ): min d(w; w 0 ) w 0 2Mod(db i) Denition of merge(db; IC). Consider DB = f< F 1 ; eval(f 1 ) > :::: < F n ; eval(f n ) >g. For i = 1:::n, let us denote eval(f i ) =< (d 1 i :::dpi i ); m i >, Then we associate DB with db 1 1 :::dbp 1 1 :::db1 n :::dbpn n where : { 8i = 1:::n, db 1 i = db2 i = ::: = dbpi i = ff i g { 8j = 1:::p i, r(db j i ) = 5 (resp; 4; 3; 2; 1) iff dj i = A (resp; B; C; D; E) merge(db; IC) is the consistent set of formulas of L so that M od(merge(db; IC)) = W;IC ([db 1 1 :::dbpn n ]) Denition of the answer to Q When addressed to DB, the answer to query Q is dened by: jjqjj DB;IC = Y ES iff j= merge(db; IC)! Q jjqjj DB;IC = N O iff j= merge(db; IC)! :Q jjqjj DB;IC =? else An alternative to answers denition. According to the approach adopted here, one can check that, in order to remain as close as possible to the recommendations, we have considered a second attribute in the evaluation of a piece of information, intending to represent its degree of inconsistency, but this attribute is not useful: the conictual aspect of an information is taken into account in the fusion process when querying the database. However, since the conictual aspect of an information seems to be important for intelligence, we suggest to modify the denition of answers as it is done in [5]. There, we suggest to associate answers with explanations as follows. Let Q be a query, the possible answers that the evaluator can generate are: { Yes, and there is no proof against it (resp, No, and there is no proof against it) when some source prove Q and no other source provide :Q.

8 { Yes, by majority (resp, No, by majority) when the number of the sources which prove Q (resp, :Q) is strictly greater than the number of the sources which provide :Q (resp, Q) { Don't know, due to a balanced inconsistency when the number of the sources which provides Q is equal to the number of the sources which provide :Q. { Dont't know, due to a lack of information when no source provide Q nor :Q. In the rst case, the explanation means that the information Q is supposed to be true and this is not conictual. In the second case, the explanation means that even if, by the majority merging process, Q is supposed to be true, this is somewhat conictual. In the third case, the explanation means that Q is highly conictual and the majority vote cannot help to decide. We do not give more detail about this alternative denition. 5 An exemple We assume a propositional language with letters a; b and c. And we assume a set of constraints IC = f:a _ :b; :a _ :cg. For instance, a could mean the observed object is a plane, b could mean the observed object is an helicopter and c could mean the altitude of the observed object is less than 3 km We consider four information sources called One; T wo; T hree and F our, whose reliability are respectively A, B, C and D. We consider the following ow of emissions: At this step, we have One emits b, then T wo emits a, then T hree emits a, then F our emits c, then One emits c. DB = f< a; < (B; C); 2 >>; < b; < (A); 2 >>; < c; < (D; A); 2 >>g The questions that raise are: is a true? is b true? is c true?... By the previous denitions, we can dene: db 1 1 = fag and r(db 1 1) = 4 db 2 1 = fag and r(db2 1 ) = 3 db 1 2 = fbg and r(db 1 2) = 5 db 1 3 = fcg and r(db 1 3) = 2 db 2 3 = fcg and r(db 2 3) = 5 The ve models of IC are: w 1 = fa; :b; :cg; w 2 = f:a; b; cg; w 3 = f:a; b; :cg, w 4 = f:a; :b; cg and w 5 = f:a; :b; :cg. Then we can compute the following distances:

9 d W (w 1 ; [db 1 1 :::db2 3]) = 12 d W (w 2 ; [db 1 1 :::db2 3]) = 7 d W (w 3 ; [db 1 1 :::db2 3]) = 14 d W (w 4 ; [db 1 1 :::db2 3]) = 12 d W (w 5 ; [db 1 1 :::db2 3 ]) = 19 Thus, M erge(db; IC) = w 2 = f:a; b; cg. Then nally, jjajj DB;IC = N O, jjbjj DB;IC = Y ES, jjcjj DB;IC = Y ES. Coming back to the intuitive meaning we gave to letters, it would mean that, given what has been registered in DB until now by the dierent sources and given their reliability, the most plausible answer is that the observed object is not a plane, it is an helicopeter which ies at less than 3km high. 6 Discussion This works intended to formalize some informal recommendations about information evaluation in information fusion. Most of the informal notions underlying the recommendations have been given a formal interpretation. However, some of them have not yet been taken into account. For instance, in this present work, we made no dierence between \being conictual" and \tending to be conictual". This last notion assumes that there is a scale for dening conicts. In the present work, this scale is binary and the formal notion which represents the conict is the logical inconsistency. Extension to graded conict is foreseen in future work. Another point which has been left aside is the total ignorance about the reliability of a given source, which was a case forseen in the recommendations. In the present formalization, it seems dicult to represent that. Indeed, which number can be attached to a source whose reliability is not known? That is an open question. However, this formalization proved that the second attribute in the evaluation of an information is not necessary. Thus, the evaluation of an information could be simplied and memorizing only the number of the sources that emitted it and their reliability is enough. As for the method of fusion, we have chosen a weighted majority because it takes into account the number of the sources and their reliability. If the number of the sources had not been so important, we could have chosen a method taking into account the reliability of the sources only (see [6], [7], [8] for instance). As for arbitration merging operator as characterized in [2] or [3], it does not consider as essential the number of sources which support an information. As for implementation issues, in order to automatically compute the answers to queries, we suggest to slightly modify the query-evaluator dened in [9] by taking into account the reliability of the sources. That extension is obvious. However, it must be recalled that this evaluator is valid only in some particular case. More specically, when 8i = 1:::n 8j = 1:::p i db i [ IC is equivalent to a set of atomic formulas, which is the case of the example given in section 5.

10 References 1. Annex to STANAG 2022 (Edition 8) North Atlantic Treaty Organization (NATO) Information Handling Services, DODSID, Issue DW S. Konieczny and R. Pino-Perez. On the logic of merging. In Principles of Knowledge Representation and Reasoning (KR'98), pages 488{498, S. Konieczny and R. Pino-Perez. Merging with Integrity Constraints In Proceedings of ESCQARU'99 4. P. Z. Revesz. On the semantics of arbitration. In International Journal of Algebra and Computation, vol 7, no 2, pp 130{160, L. Cholvy and C. Garion. Answering queries addressed to several databases according to a majority merging approach Submitted to publication. 6. C. Baral and S. Kraus and J. Minker and V.S. Subrahmanian. Combining multiple knowledge bases. IEEE Trans. on Knowledge and Data Engineering, 3(2), L. Cholvy. Proving theorems in a multi-sources environment. Proceedings of IJCAI, C. Liau. A conservative Approach to Distributed Belief Fusion Proceedings of FUSION, L. Cholvy and C. Garion. Answering queries addressed to merged databases: a query evaluator which implements a majority approach. In M-S Hacid, Z.W. Ras, D.A. Zighed, and Y. Kodrato, editors, Proc. of ISMIS-2002, LNAI n o 2366, p 131{139. Springer, june 2002.

Information Evaluation: Discussion about STANAG 2022 Recommendations

Information Evaluation: Discussion about STANAG 2022 Recommendations Laurence Cholvy ONERA Centre de Toulouse BP 4025, 2 avenue Edouard Belin 31055 Toulouse FRANCE cholvy@cert.fr Vincent Nimier ONERA Centre