Uncertainty measures on probability intervals from the imprecise Dirichlet model

Transcription

1 International Journal of General Systems, Vol. 35, No. 5, October 2006, Uncertainty measures on probability intervals from the imprecise Dirichlet model J. ABELLÁN* Department of Computer Science and Artificial Intelligence, University of Granada, Granada 18071, Spain (Received 6 February 2006; in final form 3 March 2006) When we use a mathematical model to represent information, we can obtain a closed and convex set of probability distributions, also called a credal set. This type of representation involves two types of uncertainty called conflict (or randomness) and non-specificity, respectively. The imprecise Dirichlet model (IDM) allows us to carry out inference about the probability distribution of a categorical variable obtaining a set of a special type of credal set (probability intervals). In this paper, we shall present tools for obtaining the uncertainty functions on probability intervals obtained with the IDM, which can enable these functions in any application of this model to be calculated. Keywords: Imprecise probabilities; Credal sets; Uncertainty; Entropy; Conflict; Imprecise Dirichlet model 1. Introduction Since the amount of information obtained by any action is measured by a reduction in uncertainty, the concept of uncertainty is intricately connected to the concept of information. The concept of information-based uncertainty (Klir and Wierman 1998) is related to information deficiencies such as the information being incomplete, imprecise, fragmentary, not fully reliable, vague, contradictory or deficient, and this may result in different types of uncertainty. This paper is solely concerned with the information conceived in terms of uncertainty reduction, unlike the term information as it is used in the theory of computability or in terms of logic. In classic information theory, Shannon s entropy (1948) is the tool used to quantify uncertainty. This function has certain desirable properties and has been used as the starting point when looking for another function to measure the amount of uncertainty in situations in which a probabilistic representation is not suitable. Many mathematical imprecise probability theories for representing information-based uncertainty are based on a generalization of the probability theory: e.g. Dempster Shafer s theory (DST) (Dempster 1967, Shafer 1976), interval-valued probabilities (de Campos et al. 1994), order-2 capacities (Choquet 1953/1954), upper lower probabilities (Suppes 1974, Fine 1983, * jabellan@decsai.ugr.es International Journal of General Systems ISSN print/issn online q 2006 Taylor & Francis DOI: /

2 510 J. Abellán 1988) or general convex sets of probability distributions (Good 1962, Levi 1980, Walley 1991, Berger 1994). Each of these represents a type of credal set that is a closed and convex set of probability distributions with a finite set of extreme points. In the DST, Yager (1983) distinguishes between two types of uncertainty: one is associated with cases where the information focuses on sets with empty intersections, and the other is associated with cases where the information focuses on sets where the cardinality is greater than one. These are called conflict and non-specificity, respectively. The study of uncertainty measures in the DST is the starting point for the study of these measures on more general theories. In any of these theories, it is justifiable that a measure capable of measuring the uncertainty represented by a credal set must quantify the parts of conflict and non-specificity. More recently, Abellán and Moral (2005b) and Klir and Smith (2001) justified the use of maximum entropy on credal sets as a good measure of total uncertainty. The problem lies in separating these functions into others that really do measure the conflict and non-specificity parts by using a credal set to represent the information. Abellán et al. (2006) managed to split maximum entropy into functions that are capable of coherently measuring the conflict and non-specificity of a credal set P; and also as algorithms in order to facilitate their calculation in order-2 capacities (Abellán and Moral 2005a, 2006) so that S * ðpþ ¼S * ðpþþðs * 2 S * ÞðPÞ; where S * represents maximum entropy and S * represents minimum entropy on a credal set P; with S * ðpþ coherently quantifying the conflict part of a credal set and ðs * 2 S * ÞðPÞ the nonspecificity part of a credal set. A natural way of representing knowledge is with probability intervals (Campos et al. 1994). In this paper, we shall work with a special type of probability intervals obtained using the imprecise Dirichlet model (IDM). The main use of IDM is to infer about a categorical variable. Abellán and Moral (2003b, 2005b) recently used IDM to join uncertainty measures in classification (an important problem in the field of machine learning). In this paper, we shall study IDM probability intervals and we shall prove that, while they can be represented by belief functions, they are not the only type of credal set belonging to belief functions and probability intervals. In addition, we shall present an algorithm that obtains the maximum entropy for this type of interval; we shall demonstrate a property that will enable us rapidly to obtain the minimum entropy for this type of interval; and using the fact that they represent a special type of belief function, we shall directly obtain the value of the Hartley measure on them. In Section 2 of this paper, we shall introduce the most important imprecise probability theories and distinguish between probability intervals and belief functions. In Section 3, we shall present the IDM and its main properties and shall also examine the situation of IDM probability intervals in relation to other imprecise probability theories. In Section 4, we shall explore uncertainty measures on credal sets. In Section 5, we shall outline some procedures and algorithms for obtaining the values of the main uncertainty measures on IDM probability intervals and practical examples. Conclusions are presented in Section 6.

3 2. Theories of imprecise probabilities 2.1 Credal sets All theories of imprecise probabilities that are based on classical set theory share some common characteristics (see Walley 1991, Klir 2006). One of them is that evidence within each theory is fully described by a lower probability function P * on a finite set or, alternatively, by an upper probability function P * on. These functions are always regular monotone measures (Wang and Klir 1992) that are superadditive and subadditive, respectively, and P * ð{x}þ < 1; P * ð{x}þ > 1: ð1þ x[ In the various special theories of uncertainty, they possess additional special properties. When evidence is expressed (at the most general level) in terms of an arbitrary credal set, P of probability distribution functions p, on a finite set (Kyburg 1987), functions P * and P * associated with P are determined for each set A # by the formulae P * ðaþ ¼inf pð{x}þ; P * ðaþ ¼sup pð{x}þ: ð2þ p[p x[a x[ Since for each p [ P and each A #, it follows that P * ðaþ ¼1 2 P * ð 2 AÞ: Uncertainty measures on IDM 511 p[p x[a Owing to this property, functions P * and P * are called dual (or conjugate). One of them is sufficient for capturing given evidence; the other one is uniquely determined by equation (3). It is common to use the lower probability function to capture the evidence. As is well known (Chateauneuf and Jaffray 1989, Grabisch 2000) any given lower probability function P * is uniquely represented by a set-valued function m for which mðyþ ¼0 and mðaþ ¼1; ð4þ A[ ðþ where we note () as the power set of. Any set A # for which mðaþ 0 is often called a focal element, and the set of all focal elements with the values assigned to them by function m is called a body of evidence. Function m is called a Möbius representation of P * when it is obtained for alla # via the Möbius transform mðaþ ¼ ð21þ ja2bj P * ðbþ: ð5þ BjB#A ð3þ The inverse transform is defined for all A # by the formula P * ðaþ ¼ mðbþ: BjB#A It follows directly from equation (5) P * ðaþ ¼ mðbþ; BjB>A Y ð6þ ð7þ for all A #. Assume now that evidence is expressed in terms of a given lower probability function P *. Then, the set of probability distribution functions that are consistent with P *,

4 512 J. Abellán PðP * Þ; which is always closed and convex, is defined as follows ( PðP * Þ¼ pjx [ ; pðxþ [ ½0; 1Š; pðxþ ¼1 P * ðaþ < ) pðxþ ;A # : ð8þ x[ x[a 2.2 Choquet capacities of various orders A well-defined category of theories of imprecise probabilities is based on Choquet capacities of various orders (Choquet 1953/1954). The most general theory in this category is the theory based on capacities of order 2. Here, the lower and upper probabilities, P * and P *, are monotone measures for which P * ða < BÞ > P * ðaþþp * ðbþ 2 P * ða > BÞ; P * ða > BÞ < P * ðaþþp * ðbþ 2 P * ða < BÞ; ð9þ for all A,B #. Less general uncertainty theories are then based on capacities of order k. For each k. 2, the lower and upper probabilities, P * and P *, satisfy the inequalities! [ k P * A j > j¼1 K#N k ;K Yð21Þ jkjþ1 P \ A j!; j[k! \ k P * A j < j¼1 K#N k ;K Yð21Þ jkjþ1 P [ A j!; ð10þ j[k for all families of k subsets of, where N k ¼ {1; 2;...; k}: Clearly, if k 0. k, then the theory based on capacities of order k 0 is less general than the one based on capacities of order k. The least general of all these theories is the one in which the inequalities are required to hold for all k > 2 (the underlying capacity is said to be of order 1). This theory, which was extensively developed by Shafer (1976), is usually referred to as evidence theory or DST. In this theory, lower and upper probabilities are called belief and plausibility measures, noted as Bel and Pl, respectively. An important feature of DST is that the Möbius representation of evidence m (usually called a basic probability assignment function in this theory) is a nonnegative function (m(a) [ [0,1]). Hence, we can obtain Bel and Pl function from m as the following way BelðAÞ ¼ mðbþ; PlðAÞ ¼ mðbþ: ð11þ BjB#A BjB>A Y DST is thus closely connected with the theory of random sets (Molchanov 2004). When we work with nested families of focal elements, we obtain a theory of graded possibilities, which is a generalization of classical possibility theory (De Cooman 1997, Klir 2006). 2.3 Probability intervals In this theory, lower and upper probabilities P * and P * are determined for all sets A # by intervals [l(x), u(x)] of probabilities on singletons (x [ ). Clearly, lðxþ ¼P * ð{x}þ and

5 uðxþ ¼P * ð{x}þ and inequalities (1) must be satisfied. Each given set of probability intervals I ¼ {½lðxÞ; uðxþšjx [ } is associated with a credal set, PðIÞ; of probability distribution functions, p, defined as follows ( ) PðIÞ ¼ pjx [ ; pðxþ [ ½lðxÞ; uðxþš; pðxþ ¼1 : ð12þ Sets defined in this way are clearly special cases of sets defined by equation (8). Their special feature is that they always form an (n 2 1)-dimensional polyhedron, where n ¼jj: In general, the polyhedron may have c vertices (corners), where n < c < nðn 2 1Þ; and each probability distribution function contained in the set can be expressed as a linear combination of these vertices (Weichselberger and Pöhlmann 1990, de Campos et al. 1994). A given set I of probability intervals may be such that some combinations of values taken from the intervals do not correspond to any probability distribution function. This indicates that the intervals are unnecessarily broad. To avoid this deficiency, the concept of reachability was introduced in the theory (Campos et al. 1994). A given set I is called reachable (or feasible) if and only if for each x [ and every value v(x) [ [l(x), u(x)] there exists a probability distribution function p for which pðxþ ¼vðxÞ: The reachability of any given set I can be easily checked: the set is reachable if and only if it passes the following tests lðxþþuðyþ2lðyþ < 1; ;y [ ; x[ x[ uðxþþlðyþ2uðyþ > 1; ;y [ : x[ Uncertainty measures on IDM 513 If I is not reachable, it can be converted to the set I 0 ¼ {½l 0 ðxþ; u 0 ðxþšjx [ } of reachable intervals by the formulae ( l 0 ðxþ ¼max lðxþ; 1 2 ) ; y xuðyþ ( u 0 ðxþ ¼min uðxþ; 1 2 ) ; y xlðyþ ð13þ ð14þ for all x [. Given a reachable set I of probability intervals, the lower and upper probabilities are determined for each A # by the formulae ( P * ðaþ ¼max lðxþ; 1 2 ) ; x[a xóauðxþ P * ðaþ ¼min ( uðxþ; 1 2 ) : x[a xóalðxþ ð15þ The theory based on reachable probability intervals and DST are not comparable in terms of their generalities. However, they both are subsumed under a theory based on Choquet capacities of order 2 as we can see in the following subsection.

6 514 J. Abellán 2.4 Choquet capacities of order 2 Although Choquet capacities of order 2 do not capture all credal sets, they subsume all the other special uncertainty theories that are examined in this paper. They are thus quite general. Their significance is that they are computationally easier to handle than arbitrary credal sets. In particular, it is easier to compute PðP * Þ defined by equation (8) when P * is a Choquet capacity of order 2. Let ¼ {x 1 ; x 2 ;...; x n } and let s ¼ðsðx 1 Þ; sðx 2 Þ;...; sðx n ÞÞ denote a permutation by which elements of are reordered. Then, it is established (de Campos and Bolaños 1989) that for any given Choquet capacity of order 2, PðP * Þ is determined by its extreme points, which are probability distributions p s computed as follows p s ðsðx 1 ÞÞ ¼ P * ð{sðx 1 Þ}Þ; p s ðsðx 2 ÞÞ ¼ P * ð{sðx 1 Þ; sðx 2 Þ}Þ 2 P * ð{sðx 1 Þ}Þ;... p s ðsðx n ÞÞ ¼ P * ð{sðx 1 Þ;...; sðx n Þ}Þ 2 P * ð{sðx 1 Þ;...; sðx n21 Þ}Þ: ð16þ Each permutation defines an extreme point of PðP * Þ; but different permutations can give rise to the same point. The set of distinct probability distributions p s is often called an interaction representation of P * (Grabisch 2000). Figure 1. Main uncertainty theories ordered by their generalities.

7 Uncertainty measures on IDM 515 Belief functions and reachable probability intervals represent special types of capacities of order 2, as we can see in Figure 1. However, belief functions are not generalizations of reachable probability intervals and the inverse is also not verified as we can see in Examples 1 and 2, respectively: Example 1. We consider the set ¼ {x 1 ; x 2 ; x 3 } and the following set of probability intervals on L ¼ {½0; 0:5Š; ½0; 0:5Š; ½0; 0:5Š}: This set of probability intervals L has associated a credal set, P L ; with vertices {ð0:5; 0:5; 0Þ; ð0:5; 0; 0:5Þ; ð0; 0:5; 0:5Þ}: There does not exist any basic probability assignment for this credal set. To prove this we suppose the contrary condition. Using equation (16) it can be proved that the credal set associated with a basic probability assignment on has the vertices that we can see in Table 1, where m i ¼ mð{x i }Þ; m ij ¼ mð{x i ; x j }Þ; m 123 ¼ mðþ; i; j [ {1; 2; 3}: Then, a basic probability assignment m with the same credal set, P L ; must verify that m 1 þ m 12 þ m 13 þ m 123 ¼ 0:5; m 2 þ m 12 þ m 23 þ m 123 ¼ 0:5; m 3 þ m 13 þ m 23 þ m 123 ¼ 0:5; m 1 ¼ m 2 ¼ m 3 ¼ 0; m 2 þ m 23 ¼ 0; m 3 þ m 23 ¼ 0; m 1 þ m 13 ¼ 0; m 3 þ m 13 ¼ 0; m 1 þ m 12 ¼ 0; m 2 þ m 12 ¼ 0; where any other option give us a contradiction. Hence, we have that m i ¼ 0; m ij ¼ 0(i, j [ {1,2,3}) and m 123 ¼ 0:5; implying that m is not a basic probability assignment. Example 2. We consider the following basic probability assignment m on the finite set ¼ {x 1 ; x 2 ; x 3 ; x 4 } defined by mð{x 1 ; x 2 }Þ¼0:5; mð{x 3 ; x 4 }Þ¼0:5: Table 1. Set of vertices associated with a basic probability assignment on a set of 3 elements. s p 1 p 2 p 3 (1,2,3) m 1 þ m 12 þ m 13 þ m 123 m 2 þ m 23 m 3 (1,3,2) m 1 þ m 12 þ m 13 þ m 123 m 2 m 3 þ m 23 (2,1,3) m 1 þ m 13 m 2 þ m 12 þ m 23 þ m 123 m 3 (2,3,1) m 1 m 2 þ m 12 þ m 23 þ m 123 m 3 þ m 13 (3,1,2) m 1 þ m 12 m 2 m 3 þ m 13 þ m 23 þ m 123 (3,2,1) m 1 m 2 þ m 12 m 3 þ m 13 þ m 23 þ m 123

8 516 J. Abellán Computing the upper and lower probability values for every x i, we have the following set of probability intervals compatible with m: L ¼ {½0; 0:5Š; ½0; 0:5Š; ½0; 0:5Š; ½0; 0:5Š}; but this set contains the following probability distribution p 0 ¼ð0:5; 0:5; 0; 0Þ on, that not belongs to the credal set associated with m 0 ¼ p 0 ð{x 3 ; x 4 }Þ, Bel ð{x 3 ; x 4 }Þ¼0:5; 1 ¼ p 0 ð{x 1 ; x 2 }Þ. Pl ð{x 1 ; x 2 }Þ¼0:5: However, it is easy to obtain a set of reachable probability intervals that represents the same credal set that a belief function, as we can see in the following example. Example 3. We consider the set ¼ {x 1 ; x 2 ; x 3 } and the following set of reachable probability intervals on L ¼ {½0:3; 0:65Š; ½0:2; 0:55Š; ½0:15; 0:3Š}: This set of probability intervals L has associated a credal set, P L ; with vertices {ð0:65; 0:2; 0:15Þ; ð0:3; 0:55; 0:15Þ; ð0:5; 0:2; 0:3Þ ð0:3; 0:4; 0:3Þ}: Using Table 1, it can be obtained that also this credal set is represented by the belief function associated with the basic probability assignment (has the same set of vertices) mð{x 1 }Þ¼0:3; mð{x 2 }Þ¼0:2; mð{x 3 }Þ¼0:15; mð{x 1 ; x 2 }Þ¼0:2; mð{x 1 ; x 2 ; x 3 }Þ¼0:15: 3. IDM probability intervals The IDM was introduced by Walley (1996) to draw an inference about the probability distribution of a categorical variable. Let us assume that Z is a variable taking values on a finite set and that we have a sample of size N of independent and identically distributed outcomes of Z. If we want to estimate the probabilities, u x ¼ pðxþ; with which Z takes its values, a common Bayesian procedure consists in assuming a prior Dirichlet distribution for the parameter vector (u x ) x[, and then taking the posterior expectation of the parameters given the sample. The Dirichlet distribution depends on the parameters s, a positive real value, and t, a vector of positive real numbers t ¼ðt x Þ x[ ; verifying P x[ t x ¼ 1: The density takes the form GðsÞ Y f ððu x Þ x[ Þ¼Q x[ Gðs t u s t x21 x ; xþ where G is the gamma function. If r(x) is the number of occurrences of value x in the sample, the expected posterior value of parameter u x is (r(x) þ s t x )/(), which is also the Bayesian estimate of u x (under quadratic loss). x[

9 Uncertainty measures on IDM 517 The IDM (Walley 1996) only depends on parameter s and assumes all the possible values of t. This defines a non-closed convex set of prior distributions. It represents a much weaker assumption than a precise prior model, but it is possible to make useful inferences. In our particular case, where the IDM is applied to a single variable, we obtain a credal set for this variable Z that can be represented by a system of probability intervals. For each parameter, u x, we obtain a probability interval given by the lower and upper posterior expected values of the parameter given the sample. These intervals can be easily computed and are given by [r(x)/(), (r(x) þ s)/()]. The associated credal set on is given by all the probability distributions p 0 on, such that p 0 (x) [ [r(x)/(), (r(x) þ s)/()], ;x. The intervals are coherent in the sense that if they are computed by taking infimum and supremum in the credal set, then the same set of intervals is again obtained. Parameter s determines how quickly the lower and upper probabilities converge as more data become available; larger values of s produce more cautious inferences. Walley (1996) does not give a definitive recommendation, but he advocates values between s ¼ 1 and s ¼ 2. We can define a generalization of a set of IDM probability intervals, considering that the frequencies r(x i ) are non-negative real numbers. For the sake of simplicity, we use the same name for this type of probability interval. Formally: Definition 1. Let ¼ {x 1 ;...; x n } be a finite set. Then a set of IDM probability intervals on can be defined as the set ( ) L ¼ ½l i ; u i Šjl i ¼ rðx iþ ; u i ¼ rðx iþþs ; i ¼ 1; 2;...; n; n rðx i Þ¼N ; where r(x i ) are non-negative numbers and not all are equal to zero, and s is non-negative parameter. i¼1 3.1 Properties Using the notation in definition 1, we can express the following properties: 1. Sets of IDM probability intervals generalize probability distributions. For a probability distribution p on a finite set ¼ {x 1 ;...; x n }; it is only necessary to consider s ¼ 0 and rðx i Þ¼pð{x i }Þ; for all i ¼ 1;...; n: 2. The credal set associated with a set L of IDM probability intervals, P L ; has the following set of vertices {v 1,...,v n } v 1 ¼ rðx 1Þþs ; rðx 2Þ ;...; rðx nþ v 2 ¼ rðx 1Þ ; rðx 2Þþs ;...; rðx nþ... v n ¼ rðx 1Þ ; rðx 2Þ ;...; rðx nþþs ð17þ

10 518 J. Abellán 3. Denoting as P s L the credal set associated with a set L of IDM probability intervals for a value of the parameter s and a fixed array of values r ¼ðrðx 1 Þ;...; rðx n ÞÞ; it can be verified that s 1 < s 2, P s 1 L # Ps 2 L 4. Every set of IDM probability intervals represents a set of reachable probability intervals. In Section 2.4, we see that belief functions are not generalizations of probability intervals and the inverse is also not verified. However, the credal set associated with a set of IDM probability intervals L can also be expressed by a belief function. Proposition 1. Let L be a set of IDM probability intervals as in Definition 1.The credal set associated with L is the credal set associated with the belief function associated with the basic probability assignment m L m L ð{x i }Þ¼ rðx iþ ; i ¼ 1; 2;...; n m L ðþ ¼ s m L ðaþ ¼0; ;A, ; 1, jaj, n: Proof. Using that the lower probability associated with L verifies that P * ð{x i ;...; x j }Þ¼ rðx iþþ þ rðx j Þ ; via the Möbius transform, we can obtain the following values m L ð{x i }Þ¼ rðx iþ ; m L ð{x i ; x j }Þ¼ rðx iþþrðx j Þ 2 rðx iþ 2 rðx jþ ¼ 0; m L ð{x i ; x j ; x k }Þ¼ rðx iþþrðx j Þþrðx k Þ...; 2 rðx iþþrðx j Þ 2 rðx jþþrðx k Þ 2 rðx jþþrðx k Þ þ rðx iþ þ rðx jþ þ rðx kþ ¼ 0; for all i,j,k [ {1,2,...,n}. For a general set A such that 1, jaj ¼w, n; we have m L ð{a}þ ¼ ð21þ ja2bj P * ðbþ ¼ P ð21þ ja2bj x i [B rðx iþ B#A B#A ¼ w 2 1 w 2 2 w 2 3 w 2 1 4@ A A A 2 þð21þ A5 x i [A w 2 1 ð18þ ð19þ : rðx iþ : ð20þ

11 Uncertainty measures on IDM 519 Taking into account that then 0 ¼ð1 2 1Þ w21 ¼ w 2 1 0! 2 w 2 2! þ w 2 3!! w þð21þ w21 ; ð21þ 1 3 w 2 1 Now m L ð{a}þ ¼0; m L ð{}þ ¼1 2 rðx i Þ x i ¼ s : [ ð22þ ð23þ Therefore, m L obtained is a basic probability assignment on. Now, let P L be the credal set associated with L and let P ml be the credal set associated with m L. Then, P L ¼ P ml : i) Let p [ P L be a probability distribution. Then Bel ml ðaþ ¼ P rðx i Þ < pðaþ < x i [A rðx iþþs ¼ Pl ml ðaþ; x i [A for all A #. Hence, p [ P ml ; ii) Let p [ P ml be a probability distribution. Then rðx i Þ ¼ Bel m L ð{x i }Þ < pð{x i }Þ < Pl ml ð{x i }Þ¼ rðx iþþs ; for all x i [. Hence, p [ P L : Sets of IDM probability intervals are not the only credal sets that can be expressed jointly by reachable probability intervals and belief functions. As we can observe in example 3, it is possible for a credal set to be represented by a set of reachable probability intervals and by a belief function, although this credal set cannot be represented by a set of IDM probability intervals. We only need to consider in example 3 the value s/(): it must be 0.35, using l 1 and u 1 and 0.15 using l 3 and u 3. However, the description of the credal sets belonging to reachable probability intervals and belief functions is still an open problem. In Figure 1, we can see where the sets of IDM probability intervals are placed in relation to other theories of imprecise probabilities using a generality order. A 4. An overview of uncertainty measures It has well been established that uncertainty in classical possibility theory is quantified by the Hartley measure (Hartley 1928). For each nonempty and finite set A # of possible alternatives, the Hartley measure, H(A), is defined by the formula HðAÞ ¼log 2 jaj; ð24þ

12 520 J. Abellán where jaj denotes the cardinality of A. Since HðAÞ ¼ 1 when jaj ¼ 2; H defined by equation (24) measures uncertainty in bits. The uniqueness of H was proven on axiomatic grounds by Rényi (1970). The type of uncertainty measured by H is usually called non-specificity. In classical probability theory, a justifiable measure of uncertainty was derived by Shannon (1948). This measure, which is usually referred to as Shannon entropy and denoted by S, is defined for each given probability distribution function p on a finite set by the formula SðpÞ ¼2 x[ pðxþ log 2 pðxþ: ð25þ Since SðpÞ ¼ 1 when ¼ 2 and pðxþ ¼ 1 2 pðxþ ¼ 0:5; S defined by equation (25) measures uncertainty in bits. However, the type of uncertainty measured by the Shannon entropy is different from the uncertainty type quantified by the Hartley measure; it is well captured by the term conflict. When the classical uncertainty theories are generalized, both types of uncertainty coexist. This requires the Hartley measure and Shannon entropy to be properly generalized in the various theories. The Hartley measure was first generalized for graded possibilities by Higashi and Klir (1983) and later to the DST by Dubois and Prade (1985). Its generalized form GH, is defined in terms of the Möbius representation m, by the formula GHðmÞ ¼ A#mðAÞ log 2 jaj: ð26þ The uniqueness of this generalized Hartley measure GH was proved for graded possibilities by Klir and Mariano (1987) and for the DST by Ramer (1987). Efforts to generalize the Shannon entropy to DST were less successful. Although several intuitively promising candidates for the generalized Shannon measure GS, were published in the 1980s and early 1990s, each was found to violate the essential property of subadditivity. This would have been acceptable if subadditivity were satisfied for the sum GH þ GS. Unfortunately, this was not the case for any of the proposed measures. A digest of these frustrating efforts is given in Klir and Wierman (1998) and also in Klir (2006). In the early 1990s, the unsuccessful attempts to find a generalized Shannon entropy in the DST were replaced with attempts to find an aggregated measure of both types of uncertainty (Harmanec and Klir 1994). An aggregate measure that satisfies all the required properties (additivity, subadditivity, monotonicity, proper range, etc.) was eventually found around the mid-1990s by several authors, (see Klir (2006) for more details). This aggregate uncertainty measure is a functional S * that for each belief function Bel in the DST is defined as follows S * ðbelþ ¼max P Bel ( 2 ) pðxþ log 2 pðxþ ; ð27þ x[ where the maximum is taken over the set P Bel of all probability distribution functions p that dominate the given function Bel (i.e. BelðAÞ < P x[apðxþ for all A # ). This functional can be readily generalized to any given convex set of probability distributions, as shown by Abellán and Moral (2003a). Useful algorithms for computing S * were developed for the DST by Harmanec et al. (1996), for reachable interval-valued probability distributions by Abellán

13 Uncertainty measures on IDM 521 and Moral (2003a), and for the theory based on Choquet order-2 capacities (2-monotone measures) by Abellán and Moral (2006). Although the functional S * is acceptable on mathematical grounds as an aggregate measure of uncertainty in any uncertainty theory where evidence can be represented in terms of arbitrary convex sets of probability distributions, it is highly insensitive to changes in evidence due to its aggregated nature (Klir and Smith 2001) and, moreover, it does not explicitly show measures of the two coexisting types of uncertainty, i.e. non-specificity and conflict. It is therefore desirable to disaggregate it. Clearly, S * ¼ GH þ GS; where GH and GS denote, respectively, a generalized Hartley measure (measuring non-specificity) and a generalized Shannon entropy (measuring conflict). Since S * and GH (defined by equations (26) and (27), respectively) are well established (at least in the DST), it is suggestive to define GS indirectly as the difference S * 2 GH, providing that it is non-negative. It was proven by Smith (2000) that S * 2 GH > 0 and therefore it is meaningful to take GS ¼ S * 2 GH as the generalized Shannon entropy. The disaggregated total uncertainty measure, TU, is then defined as the pair TU ¼ kgh; GSl; ð28þ where GH is defined by equation (26), S * is defined by equation (27), and GS ¼ S * 2 GH: Function GH þ GS is guaranteed to satisfy all the required mathematical properties (since GH þ GS ¼ S * Þ and it does not matter whether either of the two TU components also satisfies them. This is important since subadditivity of GH is not guaranteed beyond the DST, as demonstrated in Abellán and Moral (2005c). The idea of disaggregating S * into two components (measures of non-specificity and conflict) has opened new possibilities. One of these is based on the recognition that the following two functionals can be defined for each credal set P ( S * ðpþ ¼max 2 ) pðxþ log 2 pðxþ ; p[p x[ ( S * ðpþ ¼min 2 ) pðxþ log 2 pðxþ : p[p x[ The significance of these functionals and their difference, S * 2 S *, for capturing uncertainty associated with convex sets of probability distributions was first discussed by Kapur (1994) and Kapur et al. (1995). It was also suggested by Smith (2000) and Klir and Smith (2001). More recently, Abellán and Moral (2005a,b) further investigated properties of the difference S * 2 S * and described an algorithm for calculating the value of S *, which is applicable to any convex set of probability distributions the lower probability function of which is a Choquet order-2 capacity. They suggested that it is reasonable to view this difference as an alternative measure of non-specificity. In other words, they suggested that a measure of non-specificity, N, be defined for each credal set P of probability distributions by the formula ð29þ NðPÞ ¼S * ðpþ 2 S * ðpþ: ð30þ They also showed that functional N possesses the following properties: 1. NðPÞ [ ½0; log 2 jjš; where denotes the set of all alternatives (elementary events) on which the probability distributions in P are defined: NðPÞ ¼0 when P consists of a single

14 522 J. Abellán probability distribution; NðPÞ ¼ log 2 jj when P consists of all the probability distributions that can be defined on (total ignorance expressed by vacuous probabilities). 2. N is monotone increasing with respect to the subsethood relationship between the sets of probability distributions defined on the same set : for all i P and j P,if i P # j P then Nð i PÞ < Nð j PÞ; 3. N is continuous; 4. N is additive. These properties, which every non-specificity measure must possess, gave rise to the suggestion that this functional may be viewed as a measure of non-specificity. Unfortunately, contrary to the generalized Hartley functional, functional N violates the essential requirement of subadditivity in virtually any uncertainty theory, including the DST. This means that N is not acceptable alone as a measure of non-specificity. However, when considered as one component of a disaggregated total uncertainty measure, then the lack of subadditivity of the individual components is of no consequence; the only thing that matters is that the aggregated uncertainty S * satisfies all the essential requirements, including subadditivity. This suggests (see Abellán, et al. 2006) that an alternative disaggregated total uncertainty, a TU, be defined as the pair a TU ¼ ks * ðpþ 2 S * ðpþ; S * ðpþl: ð31þ It can be observed that the first component of a TU is the alternative non-specificity measure N, while the second component, S *, is a generalized Shannon measure (a general measure of conflict). When the two components are aggregated, we obtain S * and this functional clearly satisfies all the essential mathematical requirements. Therefore, although neither of the a TU components is subadditive, this is of no importance since the aggregated uncertainty S * is subadditive. It is interesting to observe that the functional S * has often been considered as one of the candidates for the generalized Shannon entropy. This was dismissed since neither is this subadditive nor is it subadditive when aggregated with the generalized Hartley measure GH. However, it is perfectly justifiable when aggregated with the alternative measure of nonspecificity N. In fact, some of the other candidates considered for the generalized Shannon entropy could now be considered on similar grounds, although the functional S * seems to be better justified than its competitors not only for its properties, but also for its behavior and its applicability to all credal sets. Nevertheless, viewing the measure of non-specificity in general, as the difference of the aggregate uncertainty S * and the generalized Shannon entropy GS, opens a new area of research, the purpose of which is to compare the various candidates for GS with the functional S *. 5. Computation for uncertainty measures 5.1 Upper entropy on the IDM In Abellán and Moral (2005b) we presented an algorithm to compute the upper entropy for order-2 capacities as an extension of the algorithm presented by Meyerovitz et al. (1994) for belief functions. For specific sets of probability intervals, we presented a more efficient

15 Uncertainty measures on IDM 523 algorithm in Abellán and Moral (2003a) that can be used to obtain upper entropy for a set of IDM probability intervals. Owing to the special structure of this type of interval, we can simplify the algorithm of Abellán and Moral (2003a) for the IDM. In order to express the algorithm, we shall assume that the following functions and procedures have already been implemented: Min (l) returns the index of the minimum value of the array l. Sig (l) returns the index of the second minor value of the array l. Nmin (l) returns the number of the index that attains the minimum value of the array l. Minof (a,b) returns the minimum value of the set {a,b} (real numbers). Now, let l be the array with the lower values of the set of IDM probability intervals L on a finite set ¼ {x 1 ;...; x n } such as in Definition 1, i.e. l i ¼ rðx i Þ=ðÞ: Let ^p be the array with the probability with maximum entropy. The initialization steps are: GetMaxEntro(l, ^p, s) min ˆ Min(l); sig ˆ Sig(l); nmin ˆ Nmin(l,S); For i ¼ 1ton If l i ¼ l min then s l i ˆ l i þ Minofðl sig 2 l min ; nmin Þ; ^p i ˆ l i ; s s ˆ s 2 nmin Minofðl sig 2 l min ; nmin Þ; If s. 0 then GetMaxEntro(l, ^p;s). The proof of this algorithm is the same as the one in Abellán and Moral (2003a). In the IDM applications on classification methods, such as in Abellán and Moral (2005b) or the one in Zaffalon (1999), the r(x i ) values are non-negative integer numbers and s is equal to 1 or 2. In Walley (1996), the author presents arguments which favor the value of s [ [1,2]. In addition, in Bernard (2005), new strong arguments for s ¼ 2 are presented. For these values of s [ [1,2], we can obtain the upper entropy more quickly. First, we must determine A ¼ {x j : rðx j Þ¼ min i {rðx i Þ}}: If z is the number of elements in A, then noting the distribution with maximum entropy as ^p; the algorithm GetMaxEntro can be simplified using the procedure 1. Case z. 1ors ¼ 1 8 < ^pðx i Þ¼ : rðx i Þ Nþs x i Ó A rðx i Þþs=z Nþs x i [ A: 2. Case z ¼ 1 and s. 1. Assign: r(x j ) ˆ r(x j ) þ 1 and s ˆ s 2 1, where A ¼ {x j }: Obtain new A. Obtain ^p as in case (1).

16 524 J. Abellán 5.2 Lower entropy on the IDM In Abellán and Moral (2005a), we can find an algorithm to obtain the lower entropy for an order-2 capacity. We can use this algorithm to obtain the lower entropy for a set of IDM probability intervals, but due to the special structure of this type of intervals, it can be proved that the lower entropy is quickly attained in the probability distribution obtained in the following theorem. Before this, we need the following Lemma of Wasserman and Kadane: Lemma 1(Wasserman and Kadane 1996). Let p,q be two probability distributions on a finite set ¼ {x 1 ;...; x n }: We denote p({x i }), q({x i }) as p i, q i, respectively. Let p *, q * be the same arrays arranged in a decreasing order. If P j i¼1 p* i < P j i¼1 q* i ; for j ¼ 1;...; n; then SðpÞ > SðqÞ: Theorem 1. With the above notation, let L be a set of probability intervals on. The lower entropy of P L is then obtained in the probability distribution P on P ¼ðu * 1 ; l* 2 ; l* 3 ;...; l* n Þ: ð32þ Proof. Denoting r * as the reordered array of frequencies from r in a decreasing way, p is the array r * ðx 1 Þþs ; r * ðx 2 Þ N þ S ;...; r * ðx n Þ : For any probability distribution q [ P L ; with q* being its corresponding reordered array, it can be verified that r * ðx i Þ < q* i < r * ðx i Þþs ; ;i r * ðx i Þþr * ðx j Þ r * ðx i Þþr * ðx j Þþ þ r * ðx k Þ < q * i þ q * j < r * ðx i Þþr * ðx j Þþs ; ;i; j < q * i þ q * j þ þ q * k ð33þ We can therefore obtain < r * ðx i Þþr * ðx j Þþ þ r * ðx k Þþs ;i; j;...; k: p_ 1 ¼ u * 1 ¼ r * ðx 1 Þþs > q * 1 ; p_ 1 þ p _ 2 ¼ r * ðx 1 Þþr * ðx 2 Þþs... > q * 1 þ q* 2 ; ð34þ w i¼1 p _ i ¼ P w i¼1 r * ðx i Þþs > w i¼1 q* i ; 3 < w < n: Now, using the Lemma of Wasserman and Kadane SðpÞ < SðqÞ; ;q [ P L (we should mention that it is immediate that SðpÞ ¼Sðp * Þ; for any p probability distribution). A

17 5.3 Hartley measure on the IDM This result is easy to obtain considering Property 3.1. Uncertainty measures on IDM 525 Theorem 2. Let L be a set of probability intervals on a finite variable ¼ {x 1 ;...; x n }: Then GHðP L Þ¼ s log ðnþ Proof. We only need to consider the belief function m L of Proposition 3.1. Then GHðP ml Þ¼GHðP L Þ¼ s log ðnþ: A Example 4. Using the above notation, we consider the following finite set ¼ {x 1 ; x 2 ; x 3 ; x 4 } and the vector of values r ¼ð0; 1; 10; 7Þ on. Then for s ¼ 7; the associate IDM probability intervals L is the set L ¼ 0; 7 ; ; 8 ; ; 17 ; ; Let ^p be the array of the algorithm GetMaxEntro. Then ^p has the following values on each loop of the algorithm in this order 1: ^p ¼ 0; 1 25 ; ; : ^p ¼ 1 25 ; 1 25 ; ; : ^p ¼ 3 50 ; 3 50 ; ; 7 25 where finally ^p ¼ð^pðx 1 Þ; ^pðx 2 Þ; ^pðx 3 Þ; ^pðx 4 ÞÞ in equation (3) is the probability distribution with maximum entropy of P L : The minimum value of entropy of P L is attained in the probability distribution p such that p_ ðx 1Þ¼l * 4 ¼ l 1 ¼ 0; p_ ðx 2Þ¼l * 3 ¼ l 2 ¼ 1 25 ; p_ ðx 3Þ¼u * 1 ¼ u 3 ¼ ; p_ ðx 4Þ¼l * 2 ¼ l 4 ¼ 7 25 : :

18 526 J. Abellán Now, using Theorem 2, the Hartley measure on P L is GHðP L Þ¼ s 7 log ðnþ ¼ log ð4þ: 25 Example 5. Now, we use example 7 of Abellán and Moral (2005b), where a classification problem is presented. Assume that we have a class variable with three possible values: ¼ {x 1 ; x 2 ; x 3 }: For a determinate partition of a database we have the following frequencies rðx 1 Þ¼4; rðx 2 Þ¼0; rðx 3 Þ¼0: With s ¼ 1; we have the following set of probability intervals, using the IDM 4 5 ; 1 ; 0; 1 ; 0; 1 : 5 5 Let ^p be the array of the algorithm GetMaxEntro. Now, we can use the simplification of the algorithm GetMaxEntro. We can use the case 1 of this simplification and, with the same notation, we have A ¼ {x 2 ; x 3 }; z ¼ 2 and ^p ¼ rðx 1Þ ; rðx 2Þþs=z ; rðx 3Þþs=z ¼ 4 5 ; 1 10 ; 1 10 The minimum value of entropy of P L is attained in the probability distribution p _ such that p_ ðx 1Þ¼u * 1 ¼ 1; p_ ðx 2Þ¼l * 2 ¼ 0; p_ ðx 3Þ¼l * 3 ¼ 0: Now, using Theorem 2, the Hartley measure on P L is GHðP L Þ¼ s log ðnþ ¼1 log ð3þ 5 6. Conclusions In this paper, we have presented the following study on the credal sets obtained from the probability intervals that arise when IDM is used. 1. We have proved that they represent a special type of reachable probability intervals that can be represented also by belief functions. We are determined that this type of credal sets are not the only one belongs to reachable probability intervals and belief functions. 2. We have presented their principal properties as credal sets. 3. We have developed and proved results and algorithms to obtain important uncertainty measures on this type of credal sets: maximum of entropy, minimum of entropy and Hartley measure.

19 Uncertainty measures on IDM 527 Acknowledgement This work has been supported by the Spanish Ministry of Science and Technology under the Algra project (TIN C03-02). References J. Abellán and S. Moral, Maximum of entropy for credal sets, Int. J. Uncertainty, Fuzziness Knowledge-Based Systems, 11, pp , 2003a. J. Abellán and S. Moral, Using the total uncertainty criterion for building classification trees, Int. J. Intell. Systems, 18, pp , 2003b. J. Abellán and S. Moral, Maximum difference of entropies as a non-specificity measure for credal sets, Int. J. Gen. Systems, 34, pp , 2005a. J. Abellán and S. Moral, Upper entropy of credal sets. Applications to credal classification, Int. J. Approx. Reasoning, 39, pp , 2005b. J. Abellán and S. Moral, An algorithm that computes the upper entropy for order-2 capacities, Int. J. Uncertainty, Fuzziness Knowledge-Based Systems, 14(2), pp , J. Abellán and S. Moral, Corrigendum: a non-specificity measure for convex sets of probability distributions, Int. J. Uncertainty, Fuzziness Knowledge-Based Systems, 13, p. 467, 2005c. J. Abellán, G.J. Klir and S. Moral, Disaggregated total uncertainty measure for credal sets, Int. J. Gen. Systems, 35, 2006, 35(1), pp , J.O. Berger, An overview of robust Bayesian analysis (with discussion), Test, 5, pp , J.M. Bernard, An introduction to the imprecise Dirichlet model for multinomial data, Int. J. Approx. Reasoning, 39, pp , L.M. de Campos and M.J. Bolaños, Characterization of fuzzy measures trough probabilities, Fuzzy Sets Systems, 31, pp , L.M. de Campos, J.F. Huete and S. Moral, Probability intervals: a tool for uncertainty reasoning, Int. J. Uncertainty, Fuzziness Knowledge-Based System, 2, pp , A. Chateauneuf and J.Y. Jaffray, Some characterizations of lower probabilities and other monotone capacities through the use of Möbius inversion, Math. Soc. Sci., 17, pp , G. Choquet, Théorie des Capacités, Ann. Inst. Fourier, 5, pp , 1953/1954. G. De Cooman, Possibility theor I, II, III, Int. J. Gen. Systems, 25, pp , A.P. Dempster, Upper and lower probabilities induced by a multivaluated mapping, Ann. Math. Statistic, 38, pp , D. Dubois and H. Prade, A note on measure of specificity for fuzzy sets, Int. J. Gen. Systems, 10, pp , T.L. Fine, Foundations of probability, in Basics Problems in Methodology and Linguistics, R.E. Butts and J. Hintikka, Eds., Dordrecht: Reidel, 1983, pp T.L. Fine, Lower probability models for uncertainty and nondeterministic processes, J. Stat. Plan. Infer., 20, pp , I.J. Good, Subjective probability as the measure of a non-measurable set, in Login, Methodology and Philosophy of Science, E. Nagel, P. Suppes and A. Tarski, Eds., California: Stanford University Press, 1962, pp M. Grabisch, The interaction and Möbius representations of fuzzy measures on finite speces, k-additive measures: a survey, in Fuzzy Measures and Integrals: Theory and Applications, M. Grabisch, et al. Ed., New York: Springer-Verlag, D. Harmanec and G.J. Klir, Measuring total uncertainty in Dempster-Shafer theory: a novel approach, Int. J. Gen. System, 22, pp , D. Harmanec, G. Resconi, G.J. Klir and Y. Pan, On the computation of uncertainty measure in Dempster-Shafer theory, Int. J. Gen. System, 25, pp , R.V.L. Hartley, Transmission of information, The Bell Systems Tech., 7, pp , M. Higashi and G.J. Klir, Measures of uncertainty and information based on possibility distributions, Int. J. Gen. System, 9, pp , J.N. Kapur, Measures of Information and their Applications, Ch. 23. New York: John Wiley, J.N. Kapur, G. Baciu and H.K. Kesavan, The minmax information measure, Int. J. Systems Sci., 26, pp. 1 12, G.J. Klir, Uncertainty and Information: Foundations of Generalize Information Theory, New York: John Wiley, G.J. Klir and M. Mariano, On the uniqueness of possibilistic measure of uncertainty and information, Fuzzy Sets Systems, 24, pp , G.J. Klir and R.M. Smith, On measuring uncertainty and uncertainty-based information: recent developments, Ann. Math. Artif. Intell., 32, pp. 5 33, 2001.

20 528 J. Abellán G.J. Klir and M.J. Wierman, Uncertainty-Based Information: Elements of Generalized Information Theory. Physica- Verlag/Springer-Verlag, Heidelberg and New York (2 nd ed., 1999). H.E. Kyburg, Bayesian and non-bayesian evidential updating, Artif. Intell., 31, pp , I. Levi, The Enterprise of Knowledge, London: NIT Press, A. Meyerowitz, F. Richman and E.A. Walker, Calculating maximum-entropy probabilities densities for belief functions, Int. J. Uncertainty, Fuzziness Knowledge-Based Systems, 2, pp , I. Molchanov, Theory of Random Sets, New York: Springer, A. Ramer, Uniqueness of information measure in the theory of evidence, Fuzzy Sets Systems, 35, pp , A. Rényi, Probability Theory, Amsterdam: North-Holland, G. Shafer, A Mathematical Theory of Evidence, Princeton: Princeton University Press, C.E. Shannon, A mathematical theory of communication, The Bell System Tech. J., 27, pp , , R.M. Smith, Generalized information theory: resolving some old questions and opening some new ones. PhD dissertation, Binghamton University-SUNY, Binghamton, P. Suppes, The measurement of belief (with discussion), J. Roy. Statist. Soc. B, 36, pp , P. Walley, Statistical Reasoning with Imprecise Probabilities, New York: Chapman and Hall, P. Walley, Inferences from multinomial data: learning about a bag of marbles, J. Roy. Statist. Soc. B, 58, pp. 3 57, Z. Wang and G.J. Klir, Fuzzy Measure Theory, New York: Plenum Press, L. Wasserman, J.B. Kadane, Bayesian Analisis in Statistics and Econometrics, Ch. 47. New York: John Wiley, 1996, pp K. Weichselberger and S. Pöhlmann, A Methodology for Uncertainty in Knowledge-Based Systems, New York: Springer-Verlag, R.R. Yager, Entropy and specificity in a mathematical theory of evidence, Int. J. Gen. Systems, 9, pp , M. Zaffalon, A credal approach to naive classification, Proceedings of the First International Symposium on Imprecise Probabilities and their Applications, ISIPTA 99, Gent 1999, pp Joaquín Abellán received his PhD degree in January of 2003 from the University of Granada. He is an assistant professor of computer science and artificial intelligence at the University of Granada. His current research interests are a representation of the uncertainty through convex sets of probability distributions and its applications to classification.