Information Theory over Multisets

Transcription

1 Iformatio Theory over Multisets Cosmi Bochiş, Corel Izbaşa, Gabriel Ciobau 2 Research Istitute e-austria Timişoara, Romaia {cosmi, corel}@ieat.ro 2 A.I. Cuza Uiversity, Faculty of Computer Sciece ad Romaia Academy, Istitute of Computer Sciece gabriel@ifo.uaic.ro The words, the sad words, Sometimes surroud the time As a pipe, the water which flows withi. Nichita Stăescu Summary. Startig from Shao theory of iformatio, we preset the case of producig iformatio i the form of multisets, ad ecodig iformatio usig multisets. We compute the etropy of a multiset iformatio source by costructig a equietropic strig source (with iterdepedet symbols), ad we compare this with a strig iformatio source with idepedet symbols. We the study the ecoder ad chael part of the system, obtaiig some results about multiset ecodig legth ad chael capacity. Motivatio The attempt to study iformatio sources which produce multisets istead of strigs, ad ways to ecode iformatio o multisets rather tha strigs, origiates i observig ew computatioal models like membrae systems which employ multisets [5]. Membrae systems have bee studied extesively ad there are plety of results regardig their computig power, laguage hierarchies ad complexity. However, while ay researcher workig with membrae systems (called also P systems) would agree that P systems process iformatio, ad that livig cells ad orgaisms do this too, we are uaware of ay attempt to precisely describe atural ways to ecode iformatio o multisets or to study sources of iformatio which produce multisets istead of strigs. Oe could argue that, while some of the iformatio i a livig orgaism is ecoded i a sequetial maer, like i DNA for example, there might be importat molecular iformatio sources which ivolve multisets (of molecules) i a o-trivial way. A simple questio: give a P system with, say, 2 objects a ad 3 objects b from a kow vocabulary V (suppose there are o evolutio rules), how much

2 74 C. Bochiş, C. Izbaşa, G. Ciobau iformatio is preset i that system? Also, may examples of P systems perform various computatioal tasks. Authors of such systems ecode the iput (usually umbers) i various ways, some by superimposig a strig-like structure o the membrae system [], some by usig the atural ecodig or the uary umeral system, that is, the atural umber is represeted with objects, for example, a. However, just imagie a glad which uses the bloodstream to sed molecules to some tissue which, i tur, seds back some other molecules. There is for sure a eergy ad iformatio exchage. How to describe it? Aother, more geeral way to pose that questio is: what are the atural ways to ecode umbers, ad more geerally, iformatio o multisets, ad how to measure the ecoded iformatio? If membrae systems, livig cells ad ay other (abstract or cocrete) multiset processig machies are uderstood as iformatio processig machies, the we believe that such questios should be ivestigated. Accordig to our kowledge, this is the first attempt of such a ivestigatio. We start from the idea that a study of multiset iformatio theory might produce iterestig, useful results at least i systems biology; if we uderstad the atural ways to ecode iformatio o multisets, there is a chace that Nature might be usig similar mechaisms. Aother way i which this ivestigatio seems iterestig to us is that there is more challege i efficietly ecodig iformatio o multisets, because they costitute a poorer ecodig media compared to strigs. Ecodig iformatio o strigs or eve richer, more orgaized ad complex structures are obviously possible ad have bee studied. Removig the symbol order, or their positio i the represetatio as strigs ca lead to multisets carryig a certai pealty, which deserves a precise descriptio. Order or positio do ot represet essetial aspects for iformatio ecodig; symbol multiplicity, a ative quality of multisets, is eough for may valid purposes. We focus maily o such atural approaches to iformatio ecodig over multisets, ad preset some advatages they have over approaches that superimpose a strig structure o the multiset. The we ecode iformatio usig multisets i a similar way as it is doe usig strigs. There is also a coectio betwee this work ad the theory of umeral systems. The study of umber ecodigs usig multisets ca be see as a study of a class of purely o-positioal umeral systems. 2 Etropy of a Iformatio Source Shao s iformatio theory represets oe of the great itellectual achievemets of the twetieth cetury. Iformatio theory has had a importat ad sigificat ifluece o probability theory ad ergodic theory, ad Shao s mathematics is a cosiderable ad profoud cotributio to pure mathematics. Shao s importat cotributio comes from the ivetio of the sourceecoder-chael-decoder-destiatio model, ad from the elegat ad geeral solutio of the fudametal problems which he was able to pose i terms of this model. Shao has provided sigificat demostratio of the power of codig with delay

3 Iformatio Theory over Multisets 75 i a commuicatio system, the separatio of the source ad chael codig problems, ad he has established the fudametal atural limits o commuicatio. As time goes o, the iformatio theoretic cocepts itroduced by Shao become more relevat to day-to-day more complex process of commuicatio. 2. Short Review of Shao Iformatio Theory We use the otios defied i the classical paper [6] where Shao has formulated a geeral model of a commuicatio system which is tractable to a mathematical treatmet. Defiitio. The quatity H is a reasoable measure of choice or iformatio. Cosider a iformatio source modeled by a discrete Markov process. For each possible state i of the source there is a set of probabilities (j) associated to the trasitios to state j. Each state trasitio produces a symbol correspodig to the destiatio state, e.g., if there is a trasitio from state i to state j, the symbol x j is produced. Each symbol x i has a iitial probability.. correspodig to the trasitio probability from the iitial state to each state i. We ca also view this as a radom variable X with x i as evets with probabil- ( ) x x ities, X = 2 x. p p 2 p There is a etropy H i for each state. The etropy of the source is defied as the average of these H i weighted i accordace with the probability of occurrece of the states: H(X) = i P i H i = i,j P i (j) log (j) () Suppose there are two symbols x i, x j ad p(i, j) is the probability of the successive occurrece of x i ad the x j. The etropy of the joit evet is H(i, j) = i,j p(i, j) log p(i, j) The probability of symbol x j to appear after the symbol x i is the coditioal probability (j). Strig Etropy Cosider a iformatio source which produces sequeces of symbols selected from a set of idepedet symbols x i with probabilities. The etropy formula for such a source is give i [6]: H(X) = log b

4 76 C. Bochiş, C. Izbaşa, G. Ciobau 2.2 Multiset Etropy We cosider a discrete iformatio source modeled by a discrete-time first-order Markov process (or Markov chai) which produces multiset messages (as opposed to strig messages). A message is a multiset of symbols. To compute the etropy of such a source, we costruct a equietropic source which produces strigs with mutually depedet symbols. Each strig produced by this equietropic source is a expoet of a multiset produced by the multiset source, because a multiset is a strig equivalece class. The etropy of such a source is computed by Shao s formula, where P i is the probability of state i, ad (j) is the trasitio probability from state i to state j. To compute the probability of the state i we must first observe what is specific for the multisets. The correspodig state trees are preseted i the ext figures. S 0 x x 3 x 2 7 S S 2 S 3 x x 3 x x 3 x 3 x x 2 x 2 x 2 S 4 S 5 S 6 S 7 S 8 S 9 S 0 S S 2 Fig.. Strig source states tree We take the P i for the first level of the tree, ad because P 0 = we get: P i = P 0 p 0 (i) = (2) To compute the trasitio probability (j) we kow that for multisets p(i, j) = 0 for i > j. Let N be the umber of all symbols (with repetitio allowed). The the most probable umber of symbols x j is p j N. For i j, i order to obtai j after i, we observe that the symbols x i>j caot be produced. Therefore, the probability to obtai j after i is give by the umber of favorable cases over all possible cases

5 Iformatio Theory over Multisets 77 x x 3 x 2 S S 2 S 3 x x 3 x x2 x 3 2 x 3 S 4 S 5 S 6 S 8 S 9 S 2 S 0 Fig. 2. Multiset source states tree p j N (j) = i N p j N j= = p j p j amely 0, i > j (j) = p j p, i j (3) j Theorem. The etropy formula of a multiset geeratig iformatio source is: H(X) = Proof. From, 2, ad 3 we ifer H(X) = = i,j,i j ( ) p j pj k=i p log k k=i p. (4) k ( ) p j pj k=i p log k ( ) p j pj k=i p log k k=i p. k Propositio. Whe the evets are equiprobable, i.e., =, the

6 78 C. Bochiş, C. Izbaşa, G. Ciobau H(X) = log!. Proof. We substitute for i equatio (4), ad get H(X) = = = = = ( ( ( log i + log )) i + ) i + log i + log( i + ) log i = log!. Strig Source Etropy vs. Multiset Source Etropy Theorem 2. The etropy of a multiset-producig source is lower tha or equal to the etropy of a equiprobable strig-producig source: H multiset H strig(xi equiprobable) Proof. We kow that = p j p j (5) Gibbs iequality suppose that P = {p, p 2,..., p } is a probability distributio. The for ay other probability distributio Q = {q, q 2,..., q } the followig iequality holds The log log q i (6)

7 H m (X) = (6) Iformatio Theory over Multisets 79 ( ) (5) p j pj k=i p log k k=i p k p j log p j = p j log q j with Q i = {q j where, j = i, ad q j = }, ad q j = i + : = p j log q j = ( log i + ) p j = (6) p j log p j p j log i + log( i + ) log = log = H strig (X) xi equiprobable Corollary. Whe X is equiprobable, H m H s. Proof. For = H multiset = we have log i = log! log = H strig Maximum Etropy for a Multiset Source For a multiset source, equiprobable evets do ot geerate the maximum etropy. This is obtaied by maximizig expressio 4, which seems difficult i the geeral case, but we give a example for the simplest case - with two evets (a biary multiset source): X = ( ) x x 2 p p 2 The multiset etropy for these evets is: H multiset (X) = p (p log p + p 2 log p 2 ). Let p = p H multiset (X) = p[p log p + ( p) log( p)]. Sice this fuctio has oly oe maximum i [0, ], we eed to solve: H multiset (X) = 2p[log( p) log p] log( p) = 0. A umerical ( solutio is p) The maximizig probability distributio is x X x 2 ad the maximum etropy is H multiset (X) < H strig (X equiprobable ) = log

8 80 C. Bochiş, C. Izbaşa, G. Ciobau 3 Multiset Ecodig ad Chael Capacity After explorig the characteristics of a multiset geeratig iformatio source, we move to the chael part of the commuicatio system. Properties of previously developed multiset ecodigs are aalyzed i [2, 3]. A formula for the capacity of multiset commuicatio chael is derived based o the Shao s geeral formula. Please ote that oe ca have a multiset iformatio source ad a usual sequece-based ecoder ad chael. All the followig combiatios are possible: Source/Ecoder Sequetial Multiset Sequetial [6] this paper Multiset this paper this paper Table. Source/Ecoder types 3. Strig Ecodig We shortly review the results cocerig the strig ecodig. Ecodig Legth We have a set of symbols X to be ecoded, ad a alphabet A. We cosider the uiform ecodig. Cosiderig the legth l of the ecodig, the X = {x i = a a 2... a l a j A}. If = P (x i ) =, the we have H(X) = log b() = log b () l It follows that b l. For N, b x = 0 implies x 0 l = x 0 = log b. = log b ad so Chael Capacity Defiitio 2. [6] The capacity C of a discrete chael is give by log N(T ) C = lim T T where N(T ) is the umber of allowed sigals of duratio T.

9 Iformatio Theory over Multisets 8 Theorem 3. [6] Let b (s) ij be the duratio of the s th symbol which is allowable i state i ad leads to state j. The the chael capacity C is equal to log W where W is the largest real root of the determiat equatio: where δ ij = if i = j, ad zero otherwise. 3.2 Multiset Ecodig s W b(s) ij δ ij = 0 We preset some results related to the multiset ecodig. Ecodig Legth We cosider a set X of N symbols, a alphabet A, ad the legth of ecodig l, therefore: X = {x i = a a a b b b j= j = l, a j A}. Propositio 2. No-uiform ecodigs over multisets are shorter tha uiform ecodigs over multisets. Proof. Over multisets we have ( ) b b + l (b + l )!. for a uiform ecodig: N N(b, l) = = = = l l l!(b )! b b (l + i) (x + i). If x 0 is the real root of = 0 the l = x 0. (b )! (b )! ( ) b + b + l 2. for o-uiform ecodig: N N(b +, l ) = = l l b (b + l )! i=0 (l + i) = = = l b (l + i) = l (l )!b! b! b (b )! b N(b, l). Let x 0 be the real b i=0 (x + i) root of = 0. The l = x 0. b! From N(b, x 0 ) = 0 ad x 0 b N(b, x 0 ) = 0 we get N(b, x 0) = x 0 b N(b, x 0 ). I order to prove l > l x 0 > x 0, let suppose that x 0 x 0. We have x 0 > b (for sufficietly large umbers), ad this implies that N(b, x 0 ) N(b, x 0 ) < x 0 b N(b, x 0). Sice this is false, it follows that x 0 > x 0 implies l l. Chael Capacity We cosider that a sequece of multisets is trasmitted alog the chael. The capacity of such a chael is computed for base 4, the some properties of it for ay base are preseted.

10 82 C. Bochiş, C. Izbaşa, G. Ciobau Multiset chael capacity i base 4 Fig. 3. Multiset chael capacity I Figure 3 we have a graph G(V, E) with 4 vertices V = {S, S 2, S 3, S 4 } ad E = {(i, j) i, j =..4, i j} {(i, j) i = 4, j =..3} I Theorem 3 we get b (a k) ij = t k because we cosider that the duratio to produce a k is the same for each (i, j) E. The determiat equatio is W t W t2 W t3 W t4 W tγ W t2 W t3 W t4 W tγ 0 W t3 W t4 = 0 W tγ 0 0 W t4 If we cosider t Γ = t k = t, the the equatio becomes 4 W t + 3 W 2t t W 3t = 0, ad W real = t Therefore C = log t for t =, ad so C Multiset chael capacity i base b The determiat equatio is

11 Iformatio Theory over Multisets 83 W t W t2 W t3 W t b W tγ W t2 W t3 W t b W tγ 0 W t3 W t b = W tγ 0 W t b W t b W tγ 0 0 W t b Propositio 3. If t Γ = t k = t, the the determiat equatio becomes ( W t )b = 0. (7) W t The capacity C is give by C = log b W, where W is the largest real root of the equatio (7). Cosiderig x = W t, the we have W = t C = x t log b x. (8) Sice we eed the largest real root W the we should fid the smallest positive root x of the equatio Let f b (x) = ( x) b x. ( x) b x = 0 (9) Lemma. For all b there is a uique x b (0, ) such that f b (x b ) = 0. Proof. We have f b (x) = (b )( x)b 2. b is odd f b (x) = 0 has the real root x = + k k > ad so f b (x) < 0 for all x (, ]; b is eve f b (x) 0 for all x R. Therefore f b (x) is decreasig for x (0, ), f b (0) = ad f b () =. The there exists a uique x b (0, ) such that f b (x b ) = 0. Lemma 2. The smallest positive root of Equatio (9) is decreasig with respect to b. More exactly, for all b we have x b x b+, where x b is the smallest positive root of f b (x) = 0. Proof. f b+ (x) f b (x) = ( x) b x (( x) b x) = x( x) b. The f b+ (x) f b (x) 0 for all x (0, ). Sice f b+ (x b ) 0 ad f b+ (0) =, the we have x b+ (0, x b ) accordig to Lemma. Theorem 4. Chael capacity is a icreasig fuctio with respect to b. Proof. This follows by Lemma 2 ad Equatio (8). Remark. Whe = 2, the capacity is C = t. Proof. From 2 W t = 0 we get C = log 2 t 2 = t.

12 84 C. Bochiş, C. Izbaşa, G. Ciobau 4 Coclusio Based o Shao s classical work, we preset a multiset etropy formula of a iformatio source. We also preset some relatioships betwee this etropy ad the strig etropy. For a biary multiset source, we compute a approximate maximal value for the etropy. Usig the determiat capacity formula, we compute the multiset chael capacity i base 4, ad we describe some properties of the multiset chael capacity i base b. As future work we pla to further explore the properties of multiset based commuicatio systems, ad to develop some methods for computig the maximal multiset etropy i the geeral case. A poetic visio of commuicatio Nichita Stăescu ( ) was a Romaia poet proposed for Nobel Prize for literature. Here is his view of words ad commuicatio, first i Romaia ad the i Eglish (traslatio is ours). Cuvitele / u au loc decât î cetrul lucrurilor, / umai îcojurate de lucruri. // Numele lucrurilor / u e iciodata afară. Şi totuşi / cuvitele, tristele, / îcojoară câteodata timpul / ca o ţeavă, apa care curge pri ea. //... ca şi cum ar fi lucruri..., / oho, ca şi cum ar fi lucruri... Words / do ot belog but are i the ceter of thigs, / oly surrouded by thigs. // The ames of the thigs are ever outside. But still / the words, the sad words, / sometimes surroud the time / as a pipe, the water which flows withi.//... as they would be thigs..., / oh, as they would be thigs. Nichita Stăescu For Nichita Stăescu, the pipe of words is for time what the commuicatio chael is for a message; time flows through the pipe made of words as a message passes as a fluid through that which we call a commuicatio chael. Solomo Marcus [4] Refereces. A. Ataasiu: Arithmetic with Membraes. Pre-Proceedigs of the Workshop o Multiset Processig, Curtea de Argeş, 2000, C. Bochiş, G. Ciobau, C. Izbaşa: Ecodigs ad Arithmetic Operatios i Membrae Computig. Theory ad Applicatios of Models of Computatio, Lecture Notes i Computer Sciece vol. 3959, Spriger, Berli, 2006, C. Bochiş, G. Ciobau, C. Izbaşa: Number Ecodigs ad Arithmetics over Multisets. SYNASC 06: 8th Iteratioal Symposium o Symbolic ad Numeric Algorithms for Scietific Computig. Timişoara, IEEE Computer Society, 2006,

13 Iformatio Theory over Multisets S. Marcus: Itâlirea Extremelor. Paralela 45, Bucharest, 2005, Gh. Pău: Membrae Computig. A Itroductio. Spriger, Berli, C.E. Shao: A Mathematical Theory of Commuicatio. Bell System Techical Joural, 27 (948), , ad

14