A Generalized Information Formula as the Bridge between Shannon and Popper

Transcription

1 A Generalzed Informaton Formula as the Brdge between Shannon and Popper Chenguang Lu 1 Independent Researcher, survval99@hotmal.com Abstract. A generalzed nformaton formula related to logcal probablty and fuzzy set s deduced from the classcal nformaton formula. The new nformaton measure accords wth to Popper s crteron for knowledge evoluton very much. In comparson wth square error crteron, the nformaton crteron does not only reflect error of a proposton, but also reflect the partcularty of the events descrbed by the proposton. It gves a proposton wth less logcal probablty hgher evaluaton. The paper ntroduces how to select a predcton or sentence from many for forecasts and language translatons accordng to the generalzed nformaton crteron. It also ntroduces the rate fdelty theory, whch comes from the mprovement of the rate dstorton theory n the classcal nformaton theory by replacng dstorton (.e. average error crteron wth the generalzed mutual nformaton crteron, for data compresson and communcaton effcency. Some nterestng conclusons are obtaned from the rate-fdelty functon n relaton to mage communcaton. It also dscusses how to mprove Popper s theory. 1 Introducton Although Shannon s nformaton theory s successful for electrcal communcaton, t does not deal wth semantc nformaton [1]. Semantc nformaton measures have been dscussed for long tme [], [3], [4], [5]. However, no formula can be properly used to measure the nformaton of a predcton lke Tomorrow wll be rany or Temperature s about 10 C. Twenty years ago, I set up a symmetrcal model of color vson wth four pars of opponent colors nstead of three pars as n the popular zone model [6]. To prove that the more unque colors we perceve, the hgher dscrmnaton to lghts our eyes have, and hence the more nformaton we can obtan, I researched nformaton theory. Smlarly, nformaton conveyed by color vson s also related to semantcs or meanng of symbols. From 1989 to 1993, I found the generalzed nformaton formula [7], [8] and publshed the monograph [9] on generalzed nformaton. Later, I wrote some papers for further dscusson [10], [11] and publshed a monograph on portfolo and nformaton value [1]. Ths paper wll focus on the generalzed nformaton crteron for selectng one from several sentences, or predctons, and on the rate fdelty theory, whch s an mproved verson of classcal rate dstorton theory, for data compresson and communcaton effcency. Deducng Generalzed Informaton Formula Frst we consder the nformaton provded by predctons such as The growth rate of GDP of ths year wll be 8%. Let X denote the random varable takng values from set A={x 1, x } of events, such as the growth rates or temperatures, Y denote the random varable takng values from set B={y 1, y } of sentences or predctons. For each y, there s a subset or fuzzy subset A of A and y = x A. In the classcal nformaton theory, nformaton provded by y about x s x y (1 I ( x ; y = log. x 1 see hs home page: for more artcles. 1

2 Yet, n lngustc communcaton we only know meanng of a sentence or a predcton nstead of the condton probablty x y. Fortunately, we can deduce the condton probablty x A of x whle condton x A, whch means that y = x A s true, by Bayesan formula A x x Q ( x A =, ( A where Q ( A = x A x. (3 Replacng x y wth x A n (1, we have the generalzed nformaton formula: x A A x (4 I ( x ; y = log = log, x A whch s llustrated by Fg. 1. Note that the most mportant thng s that generally x A x y because x y = x x A =x x A s reported; yet, x A = x x A =x x A s true. The y may be an ncorrect predcton or a le; yet, x A means that y must be correct. If they are always equal, then generalzed nformaton formula (4 wll become the classcal nformaton formula (1. The generalzed nformaton formula can measure not only semantc nformaton, but also sensory nformaton. Let X denote one of monochromatc lghts, Y denote the correspondng color percepton, A denote fuzzy set, whch ncludes all x that are confused wth x, of A, and A x denote the confuson probablty of x wth x. Then, a color percepton can be regarded as a sentence y = The color x s about x. Hence, the generalzed nformaton formula can also be used to measure the nformaton of a color percepton. Fg. 1. Illustraton of the generalzed nformaton formula. The more precse the predcton s, the more the nformaton s provded. Informaton mght be negatve f the predcton s obvously wrong. The greater the error We use an example to show the propertes of the formula. Assume we need to predct a stock ndex for the next weekend. Let the current ndex be x=100. There are predctons y = The ndex wll be about x and y k = The ndex wll be about x k. Assume there s pror knowledge: X x Q ( X C exp[ ( = 0 ], where C s a normalzng constant; d Q ( X x X = exp[ d 0 ( X xk ], Ak X = exp[ ]. d ( A k

3 Fgure Informaton I about stock ndex X conveyed by dfferent predctons y and y k Fgure shows the changes of nformaton conveyed by y and y k respectvely wth X changng. It tells us that the more an occasonal event s correctly predcted, the more the nformaton s. The dashed lnes show the case n whch d s reduced. The correspondng predcton may be expressed as The ndex wll be very closed to x. It can be sad that when predctons are correct, the more precse the predcton s, the more the nformaton s. If a predcton s extremely fuzzy such as The ndex wll probably go up or not go up, A X can be represented by a horzontal lne and the nformaton wll always be zero. 3 Comparng Informaton Crteron wth Square Error Crteron Assume A x s a normal functon wth the maxmum 1,.e. Q ( x x x = exp[ ], d ( A where d means the precson of a predcton or the dscrmnaton of sense organ. The less the d s, the hgher the precson or the dscrmnaton s. From (4 and (5, we have ( x x (6 I( x ; y = log A. d (5 If d and A are 1 or constants, then the nformaton wll crteron become the square error crteron. In comparson wth the square error crteron, the nformaton crteron gves more precse predctons, or predctons that predct more occasonal events, hgher evaluaton. If we use two crterons to evaluate people, the square error crteron means that no error s good; yet, the nformaton crteron means that contrbuton over error s good. Actually, phlosopher K. R. Popper suggested usng nformaton as crteron to evaluate a scentfc theory or a proposton (see page 50 n [1] long tme ago. But he ddn t provde sutable nformaton formula. The above nformaton measure accords wth Popper s theory very much [8]. If A x 1, then there must be I(x; y =0. Ths s ust the mathematcal descrpton of Popper s affrmaton that a proposton that cannot be falsfed provdes no nformaton and hence s meanngless. The less a fuzzy set A s, or the more unexpected the events n A are, the less the A s, and hence the bgger the I(x; y s whle A x =1. Ths s ust the mathematcal descrpton of Popper s affrmaton that a proposton wth less pror logcal probablty has more mportant scentfc sgnfcance f t can go though tests of facts. For sentences The temperature tomorrow mornng wll be lower than 10 C and There wll be small to medum ran tomorrow, error s hard to be expressed because there s no center x n fuzzy set A. However, measurng nformaton s the same easy for gven logcal probablty A x and probablty dstrbuton x. 3

4 4 Generalzed Kullback Formula for Sentences Selecton For gven event X= x, t s easy to select a descrptve sentence y* from many sentences y 1, y accordng to the generalzed nformaton crteron. We calculate I(x ; y for each y. The y that makes I(x ; y has the maxmum s y* we want. However, n general artfcal ntellgent systems, for gven data or evdences denoted by z, we can only know the probablty dstrbuton x z nstead of exact event x. For example, to forecast ranfall, we frst get x z accordng to observed data, and then select a sentence such as There wll be heavy ran tomorrow from many as predcton accordng to x z. In theses cases, we need generalzed Kullback formula (see Fg. : I ( X ; y ( ( = Q x A Q A x x z log = x z log, (7 x A whch s the average of I(x ; y for dfferent x. Ths formula s called generalzed Kullback formula because t has the form of Kullback formula whle x A = x z (for each. We can prove that I(X; y reaches ts maxmum when x A = x z. Now, we calculate I(X; y for dfferent y. The y that makes I(X; y have the maxmum s y* we want. We can also use the generalzed condton entropy H * ( X y = x z log x A (8 as crteron to select y *. But, actually, the calculaton s not smpler than the rght part of (7 because we need A x and A to calculate x A. For language translaton, we need to translate a sentence y n a language to another sentence y* n another language. In ths case, we need to replace x z wth x A, where A s a fuzzy subset of A, so that (7 become I ( X ; y A x = x A' log. (9 A Fg. 3. The property of the generalzed Kullback formula: the closer to the fact x z the posteror probablty x A s n comparson wth the pror probablty x, the more the nformaton about X s conveyed by y ; otherwse, the nformaton s negatve 4 Generalzed Mutual Informaton Formula Actually, the probablty x n (7 may be replaced wth subectvely forecasted probablty x so that we have I( X; y = x y x A log x (10 4

5 Calculatng the average of I(X; y for dfferent y, we have generalzed mutual nformaton formula: I( X ; X = y I ( X ; y (11 = x, y log[ x A / x ] = H ( X H ( X Y = H ( Y H ( Y Y, where H ( X x log, (1 = x H ( X Y x, y log x A, (13 = = A H ( Y y log, (14 H ( Y X x, y log A x. (15 = I call H(X forecastng entropy, whch reflects the average codng length when we economcally encode X accordng to X whle real source s X, and reaches ts mnmum as X= X. I call H(X Y posteror forecastng entropy, call H(Y generalzed entropy, and call H(Y X generalzed condton entropy or fuzzy entropy. I thnk that the generalzed nformaton s subectve nformaton and Shannon nformaton s obectve nformaton. If two weather forecasters always provde opposte forecasts and one s always correct and another s always ncorrect. They convey the same obectve nformaton, but the dfferent subectve nformaton. If X= X and X A = X y for each, whch means subectve forecasts conform to obectve facts, then the subectve mutual nformaton wll be equal to obectve or Shannon s mutual nformaton. 5 Improvng Rate Dstorton Theory nto Rate Fdelty Theory Shannon proposed the rate-dstorton functon R(D for data compresson n hs creatve paper [1]. For gven source X and the upper lmt D of dstorton d ( X, Y x, y d( x, y, (16 = where d(x, y s error measure such as square error, we change channel Y X to search the mnmum of Shannon s mutual nformaton I s (X; Y. The mnmum denoted by R=R(D s ust the rate-dstorton functon, whch reflects necessary communcaton rate for gven source X and dstorton lmt D. Actually Shannon had mentoned fdelty crteron for lossy codng. He used the dstorton,.e. average error, as the crteron for optmzng lossy codng because the fdelty crteron s hard to be formulated. However, dstorton s not a good crteron n most cases. For ths reason, I replace the error functon d =d(x, y wth generalzed nformaton I = I(x ; y and dstorton d(x, Y wth generalzed mutual nformaton I(X; Y as crteron to search the mnmum of Shannon mutual nformaton I s (X; Y for gven X=X and the lower lmt G of I(X; Y. I call ths crteron I(X; Y the fdelty crteron, call the mnmum the rate-fdelty functon R(G, and call the mproved theory the rate fdelty theory. In a way smlar to that n the classcal nformaton theory [14], we can obtan the expresson of functon R(G wth parameter s: 5

6 G( s = x y exp( si λ I (17 R( s = sg( s + x log λ where s=dr/dg ndcates the slope of functon R(G (see Fg. 3 and λ = 1/ y exp( si (18 In [1], I defned nformaton value V by the ncrement of growng speed of a portfolo because of nformaton, and suggested to use the nformaton value as crteron to optmze communcaton to get rate-value functon R(V, whch s more meanngful n some cases. 6. Propertes of Rate-fdelty Functon and Image Compresson Now we use the nformaton provded by dfferent gray levels of pxels of mages (see [9] for detals as sample to dscuss the propertes of rate-fdelty functon. The conclusons are also meanngful to lngustc communcaton. Fg. 4 Relatonshp between d and R(G for b=63 Let the gray level of a dgtzed pxel be a source and the gray level s x =, =0, 1... b = k -1 wth normal probablty dstrbuton whose expectaton=b/ and standard devaton= b/8. Assume that after decodng, the pxel also has gray level y ==0, 1... b; the percepton caused by y s also denoted by y ; and dscrmnaton functon or confusng probablty functon of x s A X = exp[ ( X /(d ] (19 where d s dscrmnaton parameter. The smaller the d s, the hgher the dscrmnaton s. Fg. 4 tells us that 1 The hgher dscrmnaton can gve us more nformaton when obectve nformaton R s bg enough; yet, lower dscrmnaton s better when obectve nformaton R s less. Ths concluson can be supported by the fact that t s better to watch TV wth less pxels or wth too much snowflake-lke dsturbance from further dstance. When R=0, G<0, whch means that f a coded mage has nothng to do wth the orgnal mage, we stll beleve t reflects the orgnal mage, then the nformaton wll be negatve. For lngustc communcaton, ths means that f one beleves a fortuneteller s talk, one would be more gnorant about facts and the nformaton he has wll be reduced. 3 When G=-, R>0, whch means that certan obectve nformaton s necessary when one uses les to deceve hs enemy to some extent; or say, les aganst facts are more terrble than les accordng to nothng. 4 The each lne of functon R(G s tangent wth the lne R=G, whch means there s a matchng pont at whch obectve nformaton s equal to subectve nformaton, and the hgher the dscrmnaton s (the less the d s, the bgger the matchng nformaton amount s. For lngustc communcaton, ths means that for mprovng effcency of communcaton, t s necessary to make obectve nformaton accord wth subectve understandng. 6

7 5 The slope of R(G becomes bgger and bgger wth G ncreasng, whch tell us that for gven dscrmnaton, t s lmted to ncrease subectve nformaton, and too much obectve nformaton s wasteful. Fg. 5. Relatonshp between matchng value of R wth G, dscrmnaton parameter d, and dgtzed bt k Fg. 5. tells us that for gven dscrmnaton, there exsts the optmal dgtzed bt k' so that the matchng value of G and R reaches the maxmum. If k<k', the matchng nformaton ncreases wth k; f k>k', the matchng nformaton no longer ncreases wth k. Ths means that too hgh resoluton of mages s unnecessary or uneconomcal for gven vsual dscrmnaton. 7 Improvement of Popper Theory Popper and hs successors tell us that relablty of a scentfc proposton comes from the repeated tests by facts. What s the dfference between the repeated tests and verfcaton emphaszed by logcal postvsm? Now we dstngush pror logcal probablty and posteror logcal probablty of a proposton. For the pror logcal probablty A, the less the better; yet for the posteror logcal probablty A x, the bgger the better. So, both falsfcaton and verfcaton are necessary. There are many probablstc and fuzzy propostons, such as Hgh humdty wll brng ran, Thrty years old s young. How do we falsfy or evaluate these propostons? Can we use a counterexample to falsfy a proposton? In theses cases, the above nformaton formula can gve these propostons approprate evaluatons. 8 Conclusons Ths paper provdes the generalzed nformaton crteron, whch accords to Popper s crteron of scentfc advance, for sentences selecton and data compresson. Its ratonalty s supported by predctons evaluaton and many propertes of rate-fdelty functon. References 1. Shannon, C. E.: A Mathematcal Theory of Communcaton. Bell System Techncal Journal, 7 ( , Weaver, W.: Recent Contrbutons to the Mathematcal Theory of Communcaton. In: The Mathematcal Theory of Communcaton, edted by C. E. Shannon and W. Weaver, Unversty of Illnos Press, Urbana ( Bar-Hllel, Y. and Carnap, R.: An Outlne of a Theory of Semantc Informaton. Tech. Rep. No.47, Research Lab. of Electroncs, MIT ( Klr, G. J. and Werman M. J.: Uncertanty-Based Informaton: Elements of Generalzed Informaton Theory (Second Edton. Physca-Verlag/Sprnger-Verlag, Hedelberg anf New York ( Zhong, Y: Prncple of Informaton Scence(n Chnese. Beng: Beng Unversty of Posts and Telecommuncatons Press ( Lu, C.: Decodng Model of Colour Vson and Verfcatons. ACTA OPTIC SINICA (n Chnese, 9( ( Lu, C.: Reform of Shannon's Formulas (n Chnese. J. of Chna Insttute of Communcaton, 1( (

8 8. Lu, C.: Coherence between the generalzed mutual nformaton formula and Popper's theory of scentfc evoluton (n Chnese. J. of Changsha Unversty, ( Lu C.: A Generalzed Informaton Theory. Chna Scence and Technology Unversty Press ( Lu, C.: Meanngs of Generalzed Entropy and Generalzed Mutual Informaton for Codng (n Chnese. J. of Chna Insttute of Communcaton, 15(6 ( Lu, C.: A Generalzaton of Shannon s Informaton Theory. Int. J.of General Systems,8(6 ( Lu, C.: Entropy Theory of Portfolo and Informaton Value. Chna Scence and Technology Unversty Press ( Popper, K. R.: Conectures and Refutatons the Growth of Scentfc Knowledge. Routledge, London and New York ( Kullback, S.: Informaton and Statstcs. John Wley & Sons Inc., New York ( Berger, T.: Rate Dstorton Theory. Englewood Clffs, N.J.: Prentce-Hall (1971 8