Uncertainty in Heuristic Knowledge and Reasoning

Transcription

1 Uncertainty in Heuristic Knowledge and Koichi YAMADA , Kami-tomioka, Nagaoka, Niigata, Japan, Management and Information Systems Science Nagaoka University of Technology Abstract: Heuristic knowledge and reasoning based on the knowledge is more or less uncertain, while computers behave logically based on rigid principles of their operation. In order for computers to mimic human intelligence using such uncertain heuristic knowledge, they must have a certain model to represent and process the uncertainty. The paper recalls how Artificial Intelligence has dealt with uncertainty; the mainstream was symbolic approaches until late 20th-century, then numerical approaches have become dominant by 21st-cenetuary. The paper summarizes major symbolic and numerical approaches with their characteristics. Then, it introduces our numerical approaches of knowledge acquisition from data and reasoning with the knowledge; a probabilistic approach, a possibilistic approach and an evidential reasoning with knowledge acquired by Rough set theory. The paper concludes with an expectation to possibilistic approaches and combined approaches of evidence theory and rough set theory. Keywords: Uncertainty, reasoning, knowledge acquisition, possibility theory, evidence theory, Dempster-Shafer theory, rough set theory 1. Introduction Uncertainty is ubiquitous in the world. We cannot spend even a day without making any decisions, and most of decisions are made under uncertain conditions to some extent. Viewed in this light, difficulties in our lives and societies seem to be mostly originated from the uncertainty in the world. Science and technology have been eliminating uncertainty due to our ignorance. However, in spite of their big success, uncertainty has not yet disappeared from the world. Instead, complexity and uncertainty we must face in our lives seem to be increasing, though science and technology are steadily being evolved. In short, our knowledge about the world will continue to be uncertain, while computers behave logically based on their rigid principles of their operation. In order for computers to mimic or simulate human intelligence using such uncertain knowledge, they must have a certain model to represent and process uncertainty. In the field of Artificial Intelligence, which aims at developing machines with functions that (are thought to) require human intelligence, uncertainty has been dealt with in various ways. Even the most basic techniques in AI such as search, logical reasoning and problem-solving could be regarded as ones to find solutions in uncertain situations. If our knowledge is certain and perfect, the solutions can easily be found without such AI techniques. Otherwise, we need some techniques of search, reasoning and/or problem-solving. In the sense, AI itself could be viewed as a field to study theories and techniques to remove or manage uncertainty in the world. In general, models and techniques to deal with uncertainty are classified into two classes: symbolic ones and numerical ones, similar to approaches of AI itself. As is well known to people in the field, AI had been dominated by symbolism until late 20-th century, where it was believed that a physical symbol system has the necessary and sufficient means for general intelligent action [1]. However, by 21st-cenetuary, several numerical approaches called computational intelligence or soft-computing such as neural networks [2], belief networks [3], fuzzy systems [4,5], evolutionary computation [6], etc. have been widely accepted as techniques to realize some kind of intelligence. In step with emerges of soft-computing techniques, numerical approaches of uncertainty have also grown up. The paper discusses uncertainty in heuristic knowledge and reasoning. In the next section, we review the vicissitudes of theories that deal with uncertainty in our knowledge and reasoning. First, a few symbol approaches are briefly reviewed. Then, numerical approaches are also discussed in some detail. Section 3 introduces our numerical approaches to acquire uncertain knowledge from data and to reason with the knowledge; a probabilistic approach, a possibilistic approach and an evidential reasoning with knowledge acquired by Rough set theory. The paper concludes with an expectation to possibilistic approaches and combined approaches of evidence theory and rough set theory. 2. Theories and Techniques for Uncertainty 2.1 Symbolic Approaches

2 In symbolic approaches, an uncertain situation is usually solved on a try and error basis; choosing an alternative among candidates (paths, rules, operations, etc.) and changing it to another if the former choice comes to a deadlock or a contradiction. This process is nothing but search, one of the basic problem-solving techniques in AI. In the sense, search would be the most primitive approach to cope with uncertainty. A well-known search technique based on uncertain knowledge is generate-and-test. It searches a state space with uncertain knowledge. A subsystem called generator, which has uncertain knowledge to choose an operation, generates a hypothesis by the chosen operation, and another subsystem called evaluator tests the hypothesis whether it satisfies given constraints. If the constraints are satisfied, the system proceeds to the next step. Otherwise, it abandons the hypothesis and generates another. When no alternative is found, the system backtracks to the previous step. The chronological backtracking used in generateand-test is computationally expensive. Thus, more elegant technique called dependency-directed backtracking was proposed [7], and is used in Truth Maintenance System [8], which is a system to preserve the whole hypotheses (called beliefs) from contradiction due to uncertain knowledge. It is sometimes called justification-based TMS, because the consistency is kept using justification possessed by each hypothesis. Assumption-based TMS proposed in [9] introduced multiple contexts to manage contradictory hypotheses in order to unnecessiate the time-consuming backtracking. In the field of binary-valued logics, a few types of non-monotonic reasoning were studied to manage uncertain knowledge. It is reasoning where some beliefs (hypotheses) may be cancelled when new knowledge is added. Default logic is one where specific type of knowledge called default rule is incorporated into a first-order predicate logic [10]. Default rule is a rule having a special condition such that "as long as the conclusion is consistent with the other beliefs." Non-monotonic logic proposed in [11,12] is a modal logic where a modal symbol is introduced to represent knowledge similar to the default rule. Circumscription introduces a meta-rule represented by 2nd-order predicate logic in order to deal with exceptions [13], which is a cause of uncertainty. A rough explanation of the meta-rule could be "when some objects satisfy a property P, only the objects satisfy the property P." For example, when penguin is concluded as an abnormal bird from some knowledge, only penguin is regarded as an abnormal bird. The idea is very similar to the closed-world assumption. The symbolic approaches were studied intensively in 70's and 80's. TMS, ATMS, or similar truth maintenance systems were incorporated into commercial software tools to develop expert systems in the mid 80's. However, there are not many reports that the functions were utilized to develop real world applications. The reasons might be that the computer capacity at the time was not enough to implement the complex algorithms, and that the knowledge base would be too complex for developers to manage large scale of applications. Non-monotonic reasoning also has problems of computability, because it needs higher-order logics. In addition, there might be issues such that human knowledge could be really represented in the framework of the theories. 2.2 Numerical Approaches One of the earliest numerical representations of uncertainty in AI might be Certainty Factor used in MYCIN, a famous medical diagnostic expert system [14]. Since then, the CF had been widely used in practical expert systems, though there were criticisms such that it is an ad hoc approach that does not have a mathematical theory, tuning of CF is a time-consuming task, etc. The reasons why probability theory was not used are 1) models of statistical reasoning at the time were too simple to be applied to rule based systems, 2) computer power at the time was too slow to implement probabilistic calculation necessary for real world problems, and 3) a very large number of data are required to get reliable probability distributions. Probabilistic reasoning has become realistic for complex applications, after reasoning and learning techniques of belief networks have been established and computer power has been powerful enough. There have been developed many applications with belief networks [3]. However, it is still a fact that many practitioners hesitate to utilize probabilistic reasoning, because it needs a large number of data to get probability distributions that give reliable reasoning results [15-17]. Possible alternatives of probability theory are Possibility theory [18-20] and Evidence theory (Dempster-Shafer theory) [5,21]. Possibility theory was initiated by by L.A.Zadeh [18], and was studied intensively by Dubois and Prade [20]. Possibility is an ordinal and qualitative scale of uncertainty when Hisdal's definition of conditional possibility [19] is used, and is insensitive to errors of prior distributions due to noises or an insufficient number of data [16,17]. Therefore, possibility might be an appropriate scale to represent uncertainty included in heuristic knowledge. Possibility has a distribution function as well as Probability has. The distribution is usually given subjectively and relatively among elements in the universal, though it can be obtained from a probability distribution utilizing a method of probability-possibility

3 transformation [17, 22]. Probability, on the other hand, is a ratio scale of uncertainty, which is quantitative and sensitive to errors of prior distributions [16,17]. Therefore, prior distributions must be given with appropriate accuracy so that the reasoning result would be within the given tolerance, even in the case that subjective probabilities are used. Probability is usually defined objectively and absolutely by frequency. Evidence theory, often referred to as Dempster- Shafer theory, is a general theory of uncertainty that includes probability and possibility as special cases. The theory differentiates two types of uncertainty included in information; non-specificity (or imprecision) and strife (or discord) [5]. Non-specificity is uncertainty related to alternatives represented by a set, and is a generalization of Hartley entropy. Strife is one to express conflicts among various sets of alternatives, and a generalization of Shannon Entropy. Uncertainty represented by Probability theory is strife. It cannot express non-specificity. Let U = {a, b, c, d} be a universal set. We have probability information only about c and d; P({c})=0.4, P({d}) = 0.2. From the information, P({a,b})=0.4 is obtained. We do not know the real probabilities of a and b. However, we must assume P({a})=0.2 and P({b})=0.2 in order to analyze the problem further using Probability theory. We certainly have knowledge that P({d})=0.2, while we must assume P({a})=0.2 and P({b})=0.2 even though we have no information about a and b. This is the meaning of comment that probability cannot express non-specificity. Possibility can represent both non-specificity and strife. However, these are severely constrained within the domain of possibility theory [5], because possibility is a special case of Evidence theory. 3. Acquiring Heuristic Knowledge with Numerical Uncertainty and The section explains three works by authors, which acquire uncertain knowledge represented by rules from data in the form of tables. The first two are knowledge acquisition from data representing causes and effects, and generate two-layered hierarchical network, where uncertainty of causes and causalities are represented by probability or possibility [15-17]. The acquired network is used to reason causes from observed effects using the theory of [23] in the case of probability, or the theory of [24,25] in the case of possibility. The third one is knowledge acquisition from multiple decision tables [26]. It utilizes an approach proposed by [27] to acquire certain rules from a decision table. Our proposal is to generate uncertain rules with basic belief assignment when multiple decision tables are given, and to generate rules that can be used in reasoning when uncertain data with bba for conditional parts is given. 3.1 Probabilistic Causal Knowledge and Please see [15] for knowledge acquisition and [23] for reasoning. 3.2 Possibilistic Causal Knowledge and Please see [17] for knowledge acquisition and [24] for reasoning. In [17], numerical experiments are conducted to compare the possibilistic approach with the probabilistic one under the condition that the number of data is too small to get reliable probability distributions. Possibilistic distributions are derived from the unreliable probability distributions using a probabilitypossibility transformation. The experiments showed 1) reasoning results with the acquired possibilistic model are comparable to those with the acquired probabilistic model as the overall evaluation, however 2) the characteristics of the reasoning results are clearly different; the reasoning results of possibilistic models are moderate in the sense that they prevent excessive ordering of uncertainty, namely many alternatives have the same rank of uncertainty, while probabilistic models give different ranks of uncertainty to all alternatives. We might conclude from the results that it is better to use possibility instead of probability to prevent excessive ordering when we cannot get reliable probability distributions. 3.3 Evidential Knowledge and [26] A. Development of Basic Evidential Mapping and Let (U, A {d}) be a decision table. U is a set of samples, A is a set of attributes and d is the decision attribute. For simplicity, let D d = {,, } be the domain of d. Y is a set of samples whose value of d is or d(u) =, u U. Then, the following rules are obtained using Rough set theory [27]. R1:if u AY, then d {}, R2:if u AY, then d {}, R3:if u AY, then d {}, R4:if u Bd (Y,Y ), then d {,}, R5:if u Bd (Y,Y ), then d {,}, R6:if u Bd (Y,Y ), then d {,}, R7:if u Bd (Y,Y,Y ), then d {,,}, where Bd (Y,Y ) = BN (Y ) BN (Y ) BN (Y ), Bd (Y,Y ) = BN (Y ) BN (Y ) BN (Y ), Bd (Y,Y ) = BN (Y ) BN (Y ) BN (Y ), Bd (Y,Y,Y ) = BN (Y ) BN (Y ) BN (Y ), BN (Y ) = AY AY, BN (Y ) = AY AY, BN (Y ) = AY AY.

4 In the above, AY, AY, BN(Y) are upper approximation, lower approximation and boundary of Y {Y,Y,Y } with respect to indiscernible relation defined by A. The conditional parts of rules R1-R7 partition and cover the sample set U. Actual rules of R1-R7 can be expressed in the following form: {e 1 } {e 2 }... {e K } Z 1 Z 2... Z H m 11 m m 1H m 21 m m 2 H... m K1 m K 2... m KH Fig. 1 Basic Evidential Mapping if a 1 (u) =x 1 and... and a N (u) = x N, then d(u) Y, (1) where a i A, x i D i and D i is the domain of attribute a i. Suppose there are multiple decision tables each of which represents decisions of an expert for the samples. In the case, the following form of rules are obtained: if a 1 (u) =x 1 and... and a N (u) = x N, then d (u) Y 1 with r 1,..., or d(u) Y M with r M, where Y j {Y,Y,Y } in the case of above example, and r j is the ratio of experts whose decision is Y j. It holds that r r M = 1, r j > 0. Thus, uncertainty in the conclusion part is expressed by basic belief assignment (bba) of Evidence theory. The rule set obtained in the above way constitutes a basic evidential mapping (BEM) proposed by [28]. See Fig. 1, where each row of the matrix represents a rule in the form of (2). e k D 1... D N represents a condition part of a rule. Z h is a subset of D d which appears in a conclusion part of rule (2). m kh is a value corresponding to r j. It is satisfied that 0 m kh 1, (2) m k m kh = 1. {e k } is called row title, and Z h is called column title. In the matrix, {e 1,...,e K } = D 1... D N must hold. So, if there is no rule for a certain e k, the following rule is added. if a(u) =e k, then d (u) D d with 1.0, (3) where a is a newly created attribute taking a value in {e 1,...,e K }. d (u) D d means completely unknown about the value of decision variable of u. For the column title, the next equations hold. {Z 1,...,Z H } = k k, (4) where k is a set of Y j,(j = 1,...M ) in the rule having a(u) = e k in its condition part. Thus, m k kj 0. Suppose the above BEM and a probability distribution on the attribute a are given. In the case, we can obtain bba on the decision attribute d by the following reasoning [28]. E 1 = {e 1 }... E K = {e K } E K +1 = {e 1,e 2 } E K +2 = {e 1,e 3 }... E K = D a Z 1 Z 2... Z H... Z H = D d m 11 m m 1H... m 1H... m K1 m K 2... m KH... m K H m K +1,1 m K +1,2... m K +1, H... m K +1, m K 1 m K 2... m K H... m K H Fig.2 Complete Evidential Mapping H m(z h ) = P(e k ) P(Z h e k ) = P(e k ) m kh, (5) k=1,k k=1,k where P(e k ) is a probability distribution on attribute a, and m kh is a conditional bba on D d given e k. m kh could also be interpreted as a conditional probability on 2 D d given e k. B. Development of Complete Evidential Mapping and Suppose the case where uncertain information on attribute a is given by bba. In the case, we need rules in the following form, in order to obtain bba on the decision attribute. if a 1 (u) X 1 and... and a N (u) X N, then d(u) Y 1 with r 1,..., or d(u) Y M with r M, (6) where X i D i. This type of rules is also necessary, when values of attributes a i are missing or cannot be specified. The rules given in the form (6) constitute a complete evidential mapping (CEM) shown in Fig.2, which has all non-empty subsets E k = X 1... X N as raw titles. If CEM and bba on attribute a are given, bba on the decision attribute can be obtained using the next equation. m(z h ) = m(e k ) m kh, (7) k=1, K where m(e k ) is bba on attribute a. Liu, et al. proposed an ad hoc way to generate CEM from BEM [28]. However, it has a problem, for example, that non-specificity of bba at raw E K +1 = {e 1,e 2 } in

5 Fig. 2 may be smaller than that at E 1 = {e 1 } or E 2 = {e 2 }. It is strange that non-specificity in conclusion part decreases even if non-specificity in condition part increases. We proposed a new way to generate CEM from BEM using a combination rule of evidence in [26]. The proposed way uses a disjunctive combination of evidence instead of a conjunctive combination such as Dempster's combination rule. CEM is generated from BEM as follows: (1) bba at raw title E k = {e k }, (k K) in CEM is the same as the one at raw title {e k } in BEM. (2) bba at raw title E k, (k > K) in CEM is obtained by the next equation. m(z h ) = m l1 (G l1 )... m l (G l ). (8) Z h =G l 1...G l G l 1,...,G l {Z 1,...,Z K } l 1,...,l {l e l E k } The equation in the above is based on the disjunctive combination of evidence shown below. m(x) = m 1 (A) m 2 (B). (9) X = AB The reason why the disjunctive combination is used is that the true information source in this case is just one (true value of attribute a is just one in E k ), while conjunctive combinations presuppose that all information sources are reliable and true [29]. 4. Conclusions Until 1990's, the main stream of uncertain reasoning was symbolic approaches. However, since they could not overcome all the difficulties residing in the real world problems, numerical approaches have emerged in step with the increase of computer powers. At present, probabilistic reasoning has been the most dominant approach to deal with uncertainty. However, probability is not perfect. There are many practitioners who hesitate to use probabilistic reasoning, because it needs a large number of data to get probability distributions that give reliable reasoning results. It is not rare that so many data is not available. The paper discussed alternative numerical approaches; Possibility theory and Evidence theory. Possibility is another scale of uncertainty that can be defined by a distribution as well as a probability. However it has quite different characteristics from probability; it is ordinal, qualitative, subjective and insensitive to errors of prior distributions. In short, probability and possibility are complementary to each other. Evidence theory was expected to be an effective alternative to theories used in Knowledge Engineering such as CF, probability, non-monotonic logic, etc. However, it could not become a major tool for real world applications due to the computational complexity. If the number of attributes in the condition part is a, and each attribute has b possible values, the number of combinations in the condition part amounts to (2 a 1) b. In the case of probability, the number is a b. Expert systems in 1970's and 80's usually have several hundreds or thousands of rules, because they needs long chaining rules and many parallel rules, which are ones having the same conclusion variable and different condition attributes. If Evidence theory is used in such expert systems, the number of rules becomes (2 a 1) b a b times by simple arithmetic. However, suppose the case where rules are acquired from from decision tables. If the number of attributes after relative reduct of Rough set theory is a, and each attribute has b possible values, the TOTAL number of rules is up to (2 a 1) b. It could be a manageable number with current computers, if the decision table is not so large. In addition, the proposed approach does not necessarily need all of the rules shown as rows in CEM. The number of rules that must be given a priori is just a b in BEM. The necessary rules, which have a combination of focal elements of bba on attribute a i in their condition parts, are usually far less than (2 a 1) b, and can be generated at the excursion time. From all those things above, evidential reasoning has become a realistic tool to manage uncertainty in the real world. References [1] A. Newell, H.A. Simon: Computer Science as Empirical Inquiry: Symbols and Search, Communications of ACM, Vol. 19, pp (1976) [2] D. E. Rumelhart, J. L. McClelland: Parallel Distributed Processing, Vol.1 & 2, MIT Press (1986) [3] J. Pearl: Probabilistic in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann (1988) [4] L. A. Zadeh: Fuzzy Sets, Information and Control, Vol.8 pp (1965) [5] G. J. Klir, B. Yuan: Fuzzy Sets and Fuzzy Logic, Theory and Applications, Prentice Hall (1995) [6] J. H. Holland: Adaptation in Natural and Artificial Systems, Univ. of Michigan Press (1975) [7] R. M. Stallman, G.J. Sussman: Forward and Dependency-directed Backtracking in a System for Computer-aided Circuit Analysis, Artificial Intelligence, Vol.9, pp (1977) [8] J. Doyle: A Truth Maintenance System, Artificial Intelligence, Vol.12, pp (1979) [9] J. de Kleer: An Assumption-Based TMS, Artificial Intelligence, Vol.28, pp (1986) [10] R. Reiter: A Logic for Default, Artificial Intelligence, Vol.13, pp (1980) [11] D. McDermott, J. Doyle: Non-monotonic Logic I, Artificial Intelligence, Vol. 13, pp (1980)

6 [12] D. McDermott: Non-monotonic Logic II: non-monotonic modal theories, Journal of the ACM, Vol. 29, pp (1982) [13] J. McCarthy: Circumscription: A Form of Non-Monotonic, Artificial Intelligence, Vol.13, pp (1980) [14] B.G.Buchanan, E.H. Shortliffe: Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project, Addison-Wesley (1984), chanan.html [15] K. Yamada: Learning causal models with conditional causal probabilities from data, Journal of Advanced Computational Intelligence, Vol.6, No.1, pp (2002) [16] K. Yamada: Possibility as An Alternative of Uncertainty Expression - Practical Advantage of Possibility over Probability, SCIS & ISIS2002, 25A2-IT5, Tsukuba (2002) [17] K. Yamada: Learning possibilistic causal model from data with transformation from probability into possibility, WSEAS Trans. on Information Science and Applications, Issue 10, Vol. 3, pp (2006) [18] L.A. Zadeh: Fuzzy sets as a basis for a theory of possibility, Fuzzy Sets and Systems, Vol.1, pp.3-28 (1978) [19] E. Hisdal: Conditional possibilities independence and non-interaction, Fuzzy Sets and Systems, Vol. 1, pp (1978) [20] D. Dubois, H. Prade: Possibility Theory, Plenum Press (1988) [21] G.A.Shafer: Mathematical Theory of Evidence, Princeton Univ. Press (1976) [22] K. Yamada: Probability - Possibility Transformation Based on Evidence Theory, IFSA World Congress 2001, pp , Vancouver (2001) [23] Y. Peng, J.A. Reggia: Abductive Inference Models for Diagnostic Problem-Solving, Springer (1990) [24] K. Yamada: Possibilistic causality consistency problem based on asymmetrically-valued causal model, Fuzzy Sets and Systems, Vol. 132/1, pp (2002) [25] K. Yamada: Diagnosis under compound effects and multiple causes by means of the conditional causal possibility approach, Fuzzy Sets and Systems, Vol. 145/2, pp (2004) [26] K. Yamada, V. Kimala, et al.: Knowledge Acquisition by Rough Sets Theory and by Evidence Theory (to appear in Japanese) [27] A. Skowron, J. Grzymala-Busse: From Rough Set Theory to Evidence Theory, in Advances in the Dempster-Shafer Theory of Evidence (eds. R.R.Yager, et al.), John Wiley & Sons, pp (1994) [28] W. Liu, et al.: Representing Heuristic Knowledge and Propagating Beliefs in the Dempster-Shafer theory of Evidence, in the same book as [27], pp (1994) [29] K. Yamada: A New Combination of Evidence Based on Compromise, Fuzzy Sets and Systems, Vol. 159/13, pp (2008)