The Entropy of Random Finite Sets

Transcription

1 The Entropy of Ranom Finite Sets Mohamma Rezaeian an Ba-Ngu Vo Department of Electrical an Electronic Engineering, University of Melbourne, Victoria, 300, Australia rezaeian, Abstract We efine the Boltzmann-Gibbs entropy of ranom finite set on a general space as the integration of logarithm of ensity function over the space of finite sets of, where the measure for this integration is the ominating measure over this space. We show that with a unit ajustment term, the same value for entropy can be obtaine using the calculus of set integrals which applies integration of ensities that have unit. Extening this concept of entropy into conitional entropy an mutual information, we use the funamental result on the relation of the mutual information an the variance of filtering error to erive a counterpart result for the signals an systems with finite set nature in a special case. While the expression for the Boltzmann- Shannon entropy of iscrete Poisson istribution is by an infinite series, we show that for a uniform Poisson ranom finite set which has Poisson carinality istribution with mean of λ, the entropy is equal to λ nat. I. INTRODUCTION In stanar systems theory, the system state is a vector that evolves in time an generates (vector) observations. Examples of this so-calle single-object system span various isciplines, ranging from econometric to biomeical engineering. Arguably, the most intuitively appealing application is in raar tracking, where the state vector contains the kinematics characteristics of a target moving in space an the observation is a raar return. The state of a wireless channel that has only one path, in conjunction with the receive training signal as observation, is another example of a single-object system. A natural generalization of a single-object system that has wie applicability to many practical problems is a multiobject system where the (multi-object) state an (multi-object) observation are finite sets of vectors [], [2]. A multi-object system is funamentally ifferent from a single-object system in that not only iniviual (vector) states evolve in time, but the number of these states also changes with time ue to birth an eaths of objects. The observation at each instant is a set of (vector) observations only some of which originate from the unerlying objects. Moreover, there is no knowlege about which object generate which observation. Primarily riven in the 970 s by applications in raar, sonar, guiance, navigation, an air traffic control (see [3] an associate references), toay, multi-object system, has foun applications in many iverse isciplines, incluing oceanography [4], robotics [5], biomeical research [6] an telecommunications [7]. In telecommunication, the multi-object system can represent the state of a multipath wireless channel. The number of paths an their vector characterization varies with time, an the receive training signal is statistically epene on this ranom set. Uncertainty in a multi-object system is characterize by moelling multi-target state an measurement as ranom finite sets, analogous to using ranom vectors for (vector) state an measurement in single-object systems. A ranom finite set is, in essence, a finite-set-value ranom variable. However, a finite set is funamentally ifferent from a vector. Stanar tools, an concepts such as probability ensity, optimal estimators, entropy etc. for ranom vectors are not irectly applicable to ranom finite sets. Point process theory is a rigorous mathematical iscipline for ealing with ranom finite sets that has long been use by statisticians in many iverse applications incluing agriculture, geology, an epiemiology [8]. Finite set statistics (FISST) is a set of practical mathematical tools from point process theory, evelope by Malher to aress multi-object systems [2]. Innovative multi-object filtering solutions erive from FISST such as the Probability Hypothesis Density (PHD) filters [2],[9],[0] have attracte substantial interest from acaemia an as well as being aopte for commercial use. Analogous to entropy for ranom vector, entropy for ranom finite set is of funamental importance in multi-object system. However, an operational notion of entropy is not yet well unerstoo for ranom finite sets. To the best of the authors knowlege, the only work in this area is that of Mahler who extene the Kullback-Leibler an Ciszar iscrimination for ranom finite sets []. Unlike a ranom vector, a ranom finite set has uncertainty in carinality (iscrete) an uncertainty in positions (continues). The entropy of a ranom finite set shoul encapsulate both of these uncertainties. In this paper we consier the measure theoretic representation of ranom finite sets base on a known ominating measure an erive the Boltzmann-Gibbz entropy [] base on this measure. Consiering a general form of representation of ensity for RFSs we erive a breakown of entropy for a RFS as the summation of components relate to various types of uncertainty for a RFS, an entropy for some known RFS will be erive. We also present a set integral formulation of entropy base on the corresponing ensity representation. Extening the efinition of entropy to conitional entropy an mutual information, we apply result of [2] to erive a relation between mutual information an mean square error for RFS estimation in a special case. In the next section we briefly iscuss various representations of RFS, as a special case of probability space, specially representation of various RFS by ensity. The above mentione contributions are iscusse in Sections III an IV.

2 II. RANDOM FINITE SETS A ranom finite set (RFS) is simply a ranom variable that take values as (unorere) finite sets, i.e. a finite-set-value ranom variable. For completeness a formal efinition of an RFS is provie in the following. A ranom finite set on R is a measurable mapping : Ω F( ) () where Ω is a sample space with a probability measure P efine on a sigma algebra of events σ(ω), an F( ) is the space of finite subsets of, which is equippe with the myopic or Matheron topology. At the funamental level, like any ranom variable, an RFS is completely escribe by its probability istribution. The probability istribution of the RFS on is the (probability) measure P on F( ) efine by P(T ) P r( T ) (2) for any Borel subset T of F( ), where T enotes the measurable subset {ω Ω : (ω) T } of Ω. In many applications it is more practical to escribe a RFS via probability ensity, rather than a probability istribution. The notion of a ensity is intimately tie to the concept of measure an integration. To escribe an RFS by a ensity, we nee to have a measure on F( ). In point process theory we use the ominating measure µ, which we efine an characterize in the following. The set of all probability ensities on a space Y, corresponing to a measure m is enote by D(Y, m) {f P/m : P (Y) }, where P/m enotes the Raon-Nykoym erivative of P relative to m. For any Borel subset of S of, let λ u (S) enote the volume measure of S in units u, an K u λ u ( ) is assume to be finite. The measure λ(a) fλ u (A) A f(x)λ u(x), where f is the constant α/k u per unit volume u, for a chosen constant α, efines the unitless Lebesgue measure which we enote by λ. For this measure λ( ) α. So α is a unitless scalar that we associate to the volume of the whole space when measure by λ. Similarly for the prouct Lebesgue measure λ r u on the space r (the rth Cartesian prouct of ), by letting f α r /Ku, r we obtain the rth prouct unitless Lebesgue measure λ r, where λ r ( r ) α r. The ominating measure µ on F( ) is efine to be µ (T ) µ (T r ), where T r is the partition of T that consists of only sets in T that have carinality r. We consier a mapping χ that maps vectors to sets by χ([x,..., x r ] T ) {x,..., x r }. For any set {x,..., x r } T r, there are vectors on r (corresponing to ifferent orerings). Therefore we measure the size T r by λ r (χ (T r ))/. Since λ r is unitless for any r, we can a up these measures for ifferent rs. Accoringly we have (using the convention 0 { }) µ (T ) λ r (χ (T ) r ). (3) Any ensity f D(F( ), µ ) efines a RFS with the probability istribution P fµ. This is similar to the conventional way of efining a continuous ranom variable on a space by a ensity f D(, λ u ). Using (3), P(T ) f(x)µ (x) T T (χ(x,..., x r ))f({x,..., x r })λ r (x...x r ), r (4) Note that P(F( ) r ) is the probability that the carinality of RFS efine by f be r, P r( r) P(F( ) r ) f({x,..., x r })λ r (x...x r ). (5) r A. Density representations of various RFSs Since λ r ( r ) α r, from (3), µ (F( )) e α, (6) therefore the simplest (uniform) ensity on F( ) is a ensity f that everywhere in F( ) is the constant e α, f({x,..., x r }) e α. (7) The RFS corresponing to this ensity is calle the uniform Poisson RFS 2. The carinality istribution of such RFS is P r( r) e α λ r (x...x r ) e α α r, (8) r which is Poisson istribution, an α is the expecte number of points for RFS, i.e: α E( ). In general a Poisson RFS is efine by f({x,..., x r }) K r ue α f (x )...f (x r ) (9) for a given f D(, λ u ), i.e: f (x)x (uniform Poisson istribution is when f (x) /K u ). The function v(x) αf (x) is calle intensity function which is sufficient to escribe Poisson RFS because v(x)x α E( ). Denoting K K u /α, ensity of (9) can be written as f({x,..., x r }) K r e α v(x )...v(x r ), an also as f({x,..., x r }) K r P (r)f (x )...f (x r ) (0) where P (r) P r( r) is Poisson istribution in (8). A more general RFS is i.i. cluster process efine by (0) in which P (r) is any arbitrary probability istribution on {0,, 2,...}. A Bernoulli RFS is when P (r) in (0) is Bernoulli over {0, }. Note that if for a ranom finite set we restrict f to be nonzero only on F( ) (the part of F( ) that represents sets of carinality one), then ue to F( ) the two concepts become ientical, an as such, a real-value ranom variable is a special case of a ranom finite set. 2 Or the ensity f({x,..., x r }) p for any 0 < p < is a uniform Poisson RFS with Poisson carinality istribution which has expectation of log p.

3 Note that a ensity epens on the measure, an in our measure µ there is one free parameter α. The parameter K in ensity (0) shows the epenency of ensity on α. Since K λu( ) λ( ) λu(s) λ(s), for any S, hence K represents the unit that we measure the space. It also makes the ensity of f unitless, because unit of f is u. The prouct form in (0) implies inepenence of occurrence of points. The most general form of ensity of a RFS that allows the occurrence of points be also epenent on each other is f({x,..., x r }) K r P (r)f r (x,..., x r ), () for a given set of symmetric ensities f r D( r, λ r u), r, 2,.... Symmetry of f means that any permutation of its arguments won t change its value, an it is zero if its arguments are not istinct. As a result, a RFS can be efine completely by a iscrete istribution P (r) an a set of symmetric ensities f r D( r, λ r u), r, 2,.... For the unitless Lebesgue measure we have for any A, λ(a) λ u (A)/K,therefore λ r (x...x r ) K r x...x r, an from (), Equation (4) reuces to P(T ) f r(x,..., x r )x...x r. χ (T ) r where f r(x,..., x r ) P (r)f r (x,..., x r ). Note that f r is not a ensity with respect to λ u, but similar to f r, it is symmetric an has unit of u r. ) Belief functional an set integral: For a close subset S, β(s) P(χ( S r )) f r(x,..., x r )x...x r S r (2) is the probability that the ranom finite set has a realization that all its points belong to S, i.e: β(s) P r( S). Defining, f ({x,.., x r }) f r(x,..., x r ) K r f({x,.., x r }) (3) The right han sie of (2) is referre to as the set integral of f, enote in general as f ()δ f ({x,..., x r })x...x r (4) S S r Note that f (like f r) has unit of u r which makes the summation in (4) possible. Accoring to (2) an (3), we have (see [3] for technical etails of proof), Theorem : For a RFS efine by the ensity function f, β(s) f ()δ, (5) S where f ({x,...x r }) K r f({x,...x r }) has unit of u r. The function β(.) is calle the belief functional of the corresponing RFS. This functional plays important role in the finite set statistics (FISST) approach to multi-target filtering introuce in by Mahler []. For moelling of multi-target system, the belief functional is more convenient than the probability istribution, since the former eals with close subsets of whereas the latter eals with subsets of F( ). The belief functional is not a measure an hence the usual notion of ensity is not applicable. In Finite set statistics (FISST), it is also shown that the f () in (5) (hence f()) can be obtaine with a reverse operation, set erivation from β(.) (see [] for further etails). Therefore a RFS can also be efine by the belief function β(s) for any S. We refer to f() as the ensity of RFS an f () K f() as the FISST probability ensity of. FISST converts the construction of multi-target ensities from multi-target moels into computing set erivatives of belief functionals. Proceures for analytically ifferentiating belief functional have been evelope an escribe in []. III. THE ENTROPY OF RFS In this section we efine the Boltzmann-Gibbs entropy for RFS an break it own to various components, an also erive a set integral relation for entropy. Measure Theoretic efinition of the Entropy For a given unitless measure m on a polish space Y, corresponing to any (unitless) ensity of f D(Y, m), the Boltzmann-Gibbs (ifferential) entropy is efine as [] h(f) η(f)m. where the function η : R + R is, { x log x x 0, η(x) 0 x 0. Y For a RFS on, with ensity f D(F( ), µ ) the Boltzmann-Gibbs entropy will be h(f) η(f)µ. (6) F( ) We refer to this expression as h(), where has ensity f. Theorem 2: For a RFS on with the general form of ensity function f in (), h() H( ) + E(h(f r )) E log(!) E( ) log K. (7) where h(f r ) f r r log(u r f r )x is the ifferential entropy 3 of the ensity f r quantity, an E log(!) in r, K K/u is a unitless P (r) log(). 3 Here for mathematical clarity, since the argument of log function shoul be a unitless value, an f r has unit of u r, we neutralize the argument by incluing unit of u r in the argument. Subsequently the last term in (7) will also have the log of a unitless value. This appearance of unit u in the expression for ifferential entropy is not essential an it is commonly implie. Alternatively, we can consier ifferential entropy for only unitless ensities f by h( f) f log( f)λ(x), where λ is the unitless Lebesgue measure. In this notation, we shoul write Eh(u r f r ) as the secon term in (7), where u r f r is unitless. Also, note that in (6) µ an f are unitless.

4 Proof: Using (3), h() η(f({x,..., x r }))λ r (x...x r ). (8) r Substituting with f in (), we have h() η(k r P (r)f r (x,..., x r ))K r x...x r r P (r) log(u r K r P (r)) P (r) f r (x,..., x r ) log(u r f r (x,..., x r ))x...x r r (9) The last summation is E(h(f r )) an the first summation breaks own to the other terms in (7). The four terms in (7) for entropy of a RFS shows various types of uncertainty for a RFS. They represent various amount of information that we require to know the realization of with certain accuracy. The first term, H( ) is the information about the number of point in the ranom set. Knowing this number is r, ientifying the position of the points with the consieration of orer, an with accuracy of almost a unit volume, for each point requires h(f r ) bits of information, an so the average of information about position of points will be E r (h(f r )). But since for ranom set orer is not important, this secon term has extra information of E(info(orer)) which must be eucte. The number of possible combinations for r points is, an given r the information about one particular combination is log( ), therefore E(info(orer)) P (r) log(). The last term in (7) is a meiator that corrects the reference of ifferential entropy to make it possible to a it to the iscrete entropy. Note that this term plus the secon term can be written as r P (r)(h(f r) log K r ), but log K r is the ifferential entropy of a uniform istribution over r, hence the ifference h(f r ) log K r corrects the reference h(f r ) to that of a uniform istribution epening on the unit measuring the space. Definition of entropy by set integral In general the set integral of a set function f over which has unit volume u is efine by (4) where the set function f () must have unit of u. Using the set integral efinition from (4) for the following function with f ({x,.., x r }) K r f({x,.., x r }), f () log(u f ())δ P (r) f r (x,..., x r )) r log(p (r)u r f r (x,..., x r ))x...x r H( ) + E(h(f r )) P (r) log() h() + E( ) log K. (20) which proves the following Theorem on the entropy by set integral. Theorem 3: For a RFS on with the FISST ensity f, h() f () log(u f ())δ E( ) log K. (2) We remark that the set integral can only be formulate on the FISST ensities f () which has unit of u, an not on the ensities f() K f () which is unitless (not to be consiere as a way to avoi the extra term E( ) log K). Relation (2) shows that the expression f () log(u f ())δ cannot be consiere as the entropy of a RFS with a FISST ensity f, but an aitive term corresponing to unit ajustment E( ) log K is also require. It is easy to show that similar to the ifferential entropy, if we change a RFS Y on R to ay, (change of unit) we nee to as such term to the entropy, i.e: h(ay ) h(y ) + E( Y ) log a. Here K K u /(uα) λu( ) uλ( ) is also a change of unit where we have change volume of from K u to α. In contrast to entropy, for mutual information an KLivergence, which are the ifference of two entropy an expectation of ratio of ensities, respectively, the ajustment term E( ) log K will be cancelle out, an the set integral can be use irectly in the efinition of these measures substituting the usual integration (see for example [, (4.64)]). Entropy of example RFSs For the i.i.. cluster point process, h(f r ) rh(f ), an the entropy reuces to h() H( ) + E( )(h(f ) log K) E log(!). (22) In particular for a Bernoulli RFS with P () q, h() h b (q) + q(h(f ) log K), where h b (.) is the binary entropy function. Also as a specialization of (22), for a Poisson RFS efine by intensity function v(.), by efining parameters α vx, f (x) v(x)/α, K u x, an substituting E( ) α, K K u /α an H( ) in (22), we have, e α α n n0 n! log e α α n n! α α log α + E log(!), (23) h() α( + h(f ) log(k u /u)) nat. In the special case of uniform Poisson RFS, h(f ) log Ku u, hence the entropy is just h() α nat. Although the entropy of the iscrete Poisson istribution in (23) is expresse by an infinite series, the entropy of a Poisson RFS is by a simple computable expression.

5 IV. CONDITIONAL ENTROPY AND MUTUAL INFORMATION Here we exten the efinition of entropy to mutual information using conitional entropy. Assuming two ranom finite sets, Y are epenent, an the conitional ensity of given realization y for Y be f x y, then h( Y y) is efine as h( y) η(f x y )µ x F( ) h( y) η(f({x,..., x r } y))λ r (x...x r ) r (24) This is a function of the set y. The conitional entropy h( Y ) is the average of this function with respect to the ensity of f y. h( Y ) t0 t! F(Y) F( ) η(f x y )f yµ x µ y t f({y,...y t })η(f({x,..., x r } {y,...y t })) λ r (x...x r )λ t (y...y t ) (25) Mutual information between the two ranom finite sets, Y is the reuction in the average resiual uncertainty about after observation of Y. I(; Y ) h() h( Y ). Relation between mutual information an estimation error In this section we exten the result of [2] on the filtering an mutual information on conventional signal processing to ranom finite sets in a special case. For, Y as ranom vectors of a vector Gaussian channel where Y γh + Z,an Z has inepenent stanar Gaussian entries, [2] has shown γ I(; Y) 2 σ2, (26) where σ 2 min g Hg(Y) H 2 f x y f yxy, an the minimizer is the conitional mean estimator. For a given ensity f x (fixe source istribution), h() will be fixe, an then in the above formulation, I(; Y) γ γ h( Y) 2 σ2. (27) In contrast to ranom variables, for ranom sets the sum of two ranom sets is not well efine. We can only efine sum for the special case that the carinality of RFS in the sum be always the same, an consier a special orering of elements. Here we consier the case Y γ + Z where, Y, Z are RFS with the constraint that the carinality of Z is always equal to the carinality of. We assume that Z has inepenent Gaussian elements an the summation is with one ranomly selecte orer, uniformly chosen. This as a multiple object system with unit probability of etection an zero probability of false alarm, but we have ranom number of objects an a noisy observation of those object as ranomize isplacement. In this case (25) can be written as h( Y ) ( )2 r f({y,...y r})η(f({x,..., x r} {y,...y r})) λ r (x...x r )λ r (y...y r ) P (r) f(y,...y r ) η(f({x,..., x r } {y,...y r })) r λ r (x...x r )λ r (y...y r ) P (r) f(y,...y r )h( r y,..., y r )λ r (y...y r ) P (r)h( r ) (28) For each r, using the result of (27), we have, γ h( r ) 2 σ2 r, r r, (29) σ 2 r min g([y,..., y r ]) g x r [x,..., x r] 2 f x y f yλ r (x...x r)λ r (y...y r) (30) an the minimizer g is the conitional mean estimator knowing r an γ. From (28) an (29), we have γ I(; Y ) 2 σ2, where σ 2 P (r)σ 2 r. (3) REFERENCES [] R. P. S. Mahler. Statistical Multisource Multitarget Information Fusion. Artech House, [2] R. P. S. Mahler, Multi-target Bayes filtering via first-orer multi-target moments, IEEE Trans. Aerospace & Electronic Systems, Vol. 39, No. 4, pp , [3] M. L. Skolnik, (E.): Raar hanbook, McGraw-Hill, 990, 2n en. [4] D. M. Lane, M. J. Chantler, an D. Dai, Robust tracking of multiple objects in sector-scan sonar image sequences using optical flow motion estimation, IEEE J. Ocean. Eng., 23, (), pp. 3 46, 998. [5] H. Durrant-Whyte an T. Bailey, Simultaneous localisation an mapping: Part I IEEE Robotics an Automation Magazine, vol. 3, no. 2,pp. 99-0, [6] B. Hammarberg, C. Forster, an E. Torebjork, Parameter estimation of human nerve C-fibers using matche filtering an multiple hypothesis tracking, IEEE Trans. Biome. Eng., 49 (4), pp , [7] E. Biglieri an M. Lops, Multiuser etection in a ynamic environment - Part I: User ientification an ata etection, IEEE Trans. Information Theory, vol. 53, no. 9, pp , Sep [8] S. Dietrich, W. Kenall, J. Mecke, Stochastic geometry an its applications, Chichester ; New York : Wiley, c995. [9] B.-N. Vo, W.-K. Ma, The Gaussian mixture Probability Hypothesis Density filter, IEEE Trans. Signal Processing, IEEE Trans. Signal Processing, Vol. 54, No., pp , [0] R. P. S. Mahler, PHD filters of higher orer in target number, IEEE Transactions on Aerospace an Electronic Systems, Vol. 43, No. 4, pp , [] W. Slomczynski. Dynamical Entropy, Markov Operators an Iterate Function Systems, Wyawnictwo Uniwersytetu Jagiellonskiego, ISBN , Krakow, [2] D. Guo, S. Shamai, S. Veru, Mutual Information an Minimum Mean- Square Error in Gaussian Channels, IEEE Transaction of Information Theory, VOL. 5, NO. 4, APRIL [3] B.-N. Vo, S. Singh, A. Doucet, Sequential Monte Carlo Methos for Multi-Target Filtering with Ranom Finite Sets, IEEE Transactions on Aerospace an Electronic Systems, VOL. 4, NO. 4 OCTOBER 2005.