On fuzzy information theory

Transcription

1 Indan Journal of Scence and Technology Vol. 3 No. 9 (Sep 200) ISSN: On fuzzy nformaton theory A. Zarand Department of athematcs & Computer, Islamc Azad unversty Kerman branch, Kerman, Iran afshn_zarand@yahoo.com Abstract Informaton theory s generally consdered to have been founded n 948 by Claude Shannon. In ths paper we ntroduce the concept of fuzzy nformaton theory usng the noton of fuzzy sets. We study t and gve some example of t. Keywords: Informaton theory, fuzzy set, fuzzy probablty, entropy. Introducton The man concepts of nformaton theory can be grasped by consderng the most wdespread means of human communcaton: language. Two mportant aspects of a concse language are as follows: Frst, the most common words (e.g., "a", "the" & "I") should be shorter than less common words (e.g., "beneft", "generaton" & "medocre"), so that sentences wll not be too long. Such a tradeoff n word length s analogous to data compresson and s the essental aspect of source codng. Second f part of a sentence s unheard or msheard due to nose e.g., a passng car the lstener should stll s able to glean the meanng of the underlyng message. Such robustness s as essental for an electronc communcaton system as t s for a language; properly buldng such robustness nto communcatons s done by channel codng. Source codng and channel codng are the fundamental concerns of nformaton theory (Reke et al.,997, Burnham & Anderson, 2002). Note that these concerns have nothng to do wth the mportance of messages. For e.g., a plattude such as "Thank you; come agan" takes about as long to say or wrte as the urgent plea, "Call an ambulance!" whle the latter may be more mportant and more meanngful n many contexts. Informaton theory, however, does not consder message mportance or meanng, as these are matters of the qualty of data rather than the quantty and readablty of data, the latter of whch s determned solely by probabltes (Anderson, 2003). Informaton theory s generally consdered to have been founded n 948 by Claude Shannon n hs semnal work, "A mathematcal theory of communcaton." (Shannon, 948; Shannon & Warren Weaver, 949) The central paradgm of classcal nformaton theory s the engneerng problem of the transmsson of nformaton over a nosy channel. The most fundamental results of ths theory are Shannon's source codng theorem whch establshes that on average the number of bts needed to represent the result of an uncertan event s gven by ts entropy; and Shannon's nosy-channel codng theorem whch states that relable communcaton s possble over nosy channels provded that the rate of communcaton s below a certan threshold called the channel capacty. The channel capacty can be approached n practce by usng approprate encodng and decodng systems. Informaton theory s closely assocated wth a collecton of pure and appled dscplnes that have been nvestgated and reduced to engneerng practce under a varety of rubrcs throughout the world over the past half century or more: adaptve systems, antcpatory systems, artfcal ntellgence, complex systems, complexty scence, cybernetcs, nformatcs, machne learnng, along wth systems scences of many descrptons. Informaton theory s a broad and deep mathematcal theory, wth equally broad and deep applcatons, amongst whch s the vtal feld of codng theory (Gbson, 998). A branch of communcaton theory devoted to problems n codng. A unque feature of nformaton theory s ts use of a numercal measure of the amount of nformaton ganed when the contents of a message are learned. Informaton theory reles heavly on the mathematcal scence of probablty. For ths reason the term nformaton theory s often appled loosely to other probablstc studes n communcaton theory, such as sgnal detecton, random nose, and predcton. See also electrcal communcatons; Probablty (Gallager, 968; Yeung, 2002). In desgnng a one-way communcaton system from the standpont of nformaton theory, three parts are consdered beyond the control of the system desgner: () the source, whch generates messages at the transmttng end of the system, (2) the destnaton, whch ultmately receves the messages, and (3) the channel, consstng of a transmsson medum or devce for conveyng sgnals from the source to the destnaton. The source does not usually produce messages n a form acceptable as nput by the channel. The transmttng end of the system contans another devce, called an encoder, whch prepares the source's messages for nput to the channel. Smlarly the recevng end of the system wll contan a decoder to convert the output of the channel nto a form that s recognzable by the destnaton. The encoder and the decoder are the parts to be desgned. In Research artcle Fuzzy nformaton theory Zarand Indan Socety for Educaton and Envronment (See) Indan J.Sc.Technol.

2 Indan Journal of Scence and Technology Vol. 3 No. 9 (Sep 200) ISSN: rado systems ths desgn s essentally the choce of a modulator and a detector. See also odulaton. A source s called dscrete f ts messages are sequences of elements (letters) taken from an enumerable set of possbltes (alphabe. Thus sources producng nteger data or wrtten Englsh are dscrete. Sources whch are not dscrete are called contnuous, for example, speech and musc sources. The treatment of contnuous cases s sometmes smplfed by notng that sgnal of fnte bandwdth can be encoded nto a dscrete sequence of numbers. The output of a channel need not agree wth ts nput. For e.g., a channel mght, for secrecy purposes, contan a cryptographc devce to scramble the message. Stll, f the output of the channel can be computed knowng just the nput message, then the channel s called noseless. If, however, random agents make the output unpredctable even when the nput s known, then the channel s called nosy (Reza, 994). any encoders frst break the message nto a sequence of elementary blocks; next they substtute for each block a representatve code, or sgnal, sutable for nput to the channel. Such encoders are called block encoders. For e.g., telegraph and teletype systems both use block encoders n whch the blocks are ndvdual letters. Entre words form the blocks of some commercal cablegram systems. It s generally mpossble for a decoder to reconstruct wth certanty a message receved va a nosy channel. Sutable encodng, however, may make the nose tolerable (Csszar & Janos, 997). Even when the channel s noseless a varety of encodng schemes exsts and there s a problem of pckng a good one. Of all encodngs of Englsh letters nto dots and dashes, the Contnental orse encodng s nearly the fastest possble one. It acheves ts speed by assocatng short codes wth the most common letters. A noseless bnary channel (capable of transmttng two knds of pulse 0,, of the same duraton) provdes the followng e.g. suppose one had to encode Englsh text for ths channel. A smple encodng mght just use 27 dfferent fve-dgt codes to represent word space (denoted by #), A, B,, Z; say # 00000, A 0000, B 0000, C 000,, Z 0. The word #CAB would then be encoded nto A smlar encodng s used n teletype transmsson; however, t places a thrd knd of pulse at the begnnng of each code to help the decoder stay n synchronsm wth the encoder (Goldman, 2005). Informaton theory s based on probablty theory and statstcs. The most mportant quanttes of nformaton are entropy, the nformaton n a random varable and mutual nformaton, the amount of nformaton n common between two random varables. The former quantty ndcates how easly message data can be compressed whle the latter can be used to fnd the communcaton rate across a channel. The choce of logarthmc base n the followng formulae determnes the unt of nformaton entropy that s used. The most common unt of nformaton s the bt, based on the bnary logarthm. Other unts nclude the nat, whch s based on the natural logarthm, and the hartley, whch s based on the common logarthm. In what follows, an expresson of the form s consdered by conventon to be equal to zero whenever p = 0. Ths s justfed because for any logarthmc base (Jaynes 957, ackay 2003). Entropy Entropy of a Bernoull tral as a functon of success probablty often called the bnary entropy functon H b ( p). The entropy s maxmzed at bt per tral when the two possble outcomes are equally probable, as n an unbased con toss (Kolmogorov,968). The entropy, H, of a dscrete random varable s a measure of the amount of uncertanty assocated wth the value of. Suppose one transmts 000 bts (0s & s). If these bts are known ahead of transmsson (to be a certan value wth absolute probablt, logc dctates that no nformaton has been transmtted. If, however, each s equally and ndependently lkely to be 0 or, 000 bts (n the nformaton theoretc sense) have been transmtted. Between these two extremes, nformaton can be quantfed as follows. If s the set of all messages { x,... x n } that could be, and x) s the probablty of gven some, then the entropy of s defned: H ( ) = x x)log x) Here, I(x) s the self-nformaton, whch s the entropy contrbuton of an ndvdual message and s the expected value. An mportant property of entropy s that t s maxmzed when all the messages n the message space are equprobable x) = / n,.e., most unpredctable n H( ) = logn. whch case The specal case of nformaton entropy for a random varable wth two outcomes s the bnary entropy functon, usually taken to the logarthmc base 2. To recaptulate, we assume the followng four condtons as axoms: - H (,,..., ) = f ( ) s a monotoncally ncreasng functon of (=, 2, ). 2- f ( L) = f ( ) + f ( L) (, L=, 2, ) 3- H ( p, p,..., p 2 ) = p pr H ( p + p pr, pr p ) + ( p pr ) H (,..., ) r r p p pr+ p + ( p r p ) H (,..., p p = r+ = r+ ) = = Research artcle Fuzzy nformaton theory Zarand Indan Socety for Educaton and Envronment (See) Indan J.Sc.Technol.

3 Indan Journal of Scence and Technology Vol. 3 No. 9 (Sep 200) ISSN: H( p, p) s a contnuous functon of p. The four axoms essentally determne the uncertanty measure. ore precsely, we have the followng theorem. Theorem. (Yeung 2008) The only functon satsfyng the four gven axoms s H ( p,..., p ) = C p log p = Where, C s an arbtrary postve number and the logarthm base s any number greater than. Unless otherwse specfed, we shall assume C= and take logarthms to the base 2. The unts of H are sometmes called bts (a contracton of bnary dgts). Thus the unts are chosen so that there s one bt of uncertanty assocated wth the toss of an unbased con. Basng the con tends to decrease the uncertanty. We remark n passng that the average uncertanty of a random varable does not depend on the values the random varable assumes or on anythng else except the probabltes assocated wth those values. The average uncertanty assocated wth the toss of an unbased con s not changed by addng the condton that the expermenter wll be shot f the con comes up tals. Jont entropy The jont entropy of two dscrete random varables and Y s merely the entropy of ther parng: (, Y). Ths mples that f and Y are ndependent, then ther jont entropy s the sum of ther ndvdual entropes. For e.g, f (, Y) represents the poston of a chess pece the row and Y the column, then the jont entropy of the row of the pece and the column of the pece wll be the entropy of the poston of the pece. H (, Y) = y log Despte smlar notaton, jont entropy should not be confused wth cross entropy. Condtonal entropy (equvocaton) The condtonal entropy or condtonal uncertanty of gven random varable Y (also called the equvocaton of about Y) s the average condtonal entropy over Y: H ( Y) = log y Because entropy can be condtoned on a random varable or on that random varable beng a certan value, care should be taken not to confuse these two defntons of condtonal entropy, the former of whch s n more common use. A basc property of ths form of condtonal entropy s that (Gallager 968): H ( Y ) = H (, Y ) H ( Y ) utual nformaton (transnformaton) utual nformaton measures the amount of nformaton that can be obtaned about one random varable by observng another. It s mportant n communcaton where t can be used to maxmze the amount of nformaton shared between sent and receved sgnals. The mutual nformaton of relatve to Y s gven by: I(, Y) = log x y ) Where, SI (Specfc mutual Informaton) s the pont wse mutual nformaton. A basc property of the mutual nformaton s that I(, Y ) = H ( ) H ( Y ) That s, knowng Y, we can save an average of I(; Y) bts n encodng compared to not knowng Y. utual nformaton s symmetrc: I( Y ) = I( Y ) = H ( ) + H ( Y ) H (, Y ) utual nformaton can be expressed as the average Kullback Lebler dvergence (nformaton gan) of the posteror probablty dstrbuton of gven the value of Y to the pror dstrbuton on. I( ; Y ) = E p ( [ DKL ( Y = ))] In other words, ths s a measure of how much, on the average, the probablty dstrbuton on wll change f we are gven the value of Y. Ths s often recalculated as the dvergence from the product of the margnal dstrbutons to the actual jont dstrbuton: I( ; Y) = D (, Y ) ) Y )) KL utual nformaton s closely related to the loglkelhood rato test n the context of contngency tables and the multnomal dstrbuton and to Pearson's χ2 test: mutual nformaton can be consdered a statstc for assessng ndependence between a par of varables, and has a well-specfed asymptotc dstrbuton. Kullback Lebler dvergence (nformaton gan) The Kullback Lebler dvergence (or nformaton dvergence, nformaton gan, or relatve entrop s a way of comparng two dstrbutons: a "true" probablty dstrbuton p (), and an arbtrary probablty dstrbuton q (). If we compress data n a manner that assumes q () s the dstrbuton underlyng some data, when, n realty, p () s the correct dstrbuton, the Kullback Lebler dvergence s the number of average addtonal bts per datum necessary for compresson. It s thus defned x) DKL ( ) q( Y )) = x)log x q( x) Although t s sometmes used as a 'dstance metrc', t s not a true metrc snce t s not symmetrc and does not Research artcle Fuzzy nformaton theory Zarand Indan Socety for Educaton and Envronment (See) Indan J.Sc.Technol.

4 Indan Journal of Scence and Technology Vol. 3 No. 9 (Sep 200) ISSN: satsfy the trangle nequalty (makng t a semquasmetrc) (ansurpur, 987). Other quanttes Other mportant nformaton theoretc quanttes nclude Rény entropy, (a generalzaton of entropy,) dfferental entropy, (a generalzaton of quanttes of nformaton to contnuous dstrbutons,) and the condtonal mutual nformaton. Codng theory Codng theory s one of the most mportant and drect applcatons of nformaton theory. It can be subdvded nto source codng theory and channel codng theory. Usng a statstcal descrpton for data, nformaton theory quantfes the number of bts needed to descrbe the data, whch s the nformaton entropy of the source. Data compresson (source codng): There are two formulatons for the compresson problem: lossless data compresson: the data must be reconstructed exactly; lossy data compresson: allocates bts needed to reconstruct the data, wthn a specfed fdelty level measured by a dstorton functon. Ths subset of Informaton theory s called rate dstorton theory. Errorcorrectng codes (channel codng): Whle data compresson removes as much redundancy as possble, an error correctng code adds just the rght knd of redundancy (.e., error correcton) needed to transmt the data effcently and fathfully across a nosy channel. Ths dvson of codng theory nto compresson and transmsson s justfed by the nformaton transmsson theorems, or source channel separaton theorems that justfy the use of bts as the unversal currency for nformaton n many contexts. However, these theorems only hold n the stuaton where one transmttng user wshes to communcate to one recevng user. In scenaros wth more than one transmtter (the multpleaccess channel), more than one recever (the broadcast channel) or ntermedary "helpers" (the relay channel), or more general networks, compresson followed by transmsson may no longer be optmal. Network nformaton theory refers to these mult-agent communcaton models. Source theory Any process that generates successve messages can be consdered a source of nformaton. A memory-less source s one n whch each message s an ndependent dentcally-dstrbuted random varable, whereas the propertes of ergodcty and statonarty mpose more general constrants. All such sources are stochastc. These terms are well studed n ther own rght outsde nformaton theory. Rate Informaton rate s the average entropy per symbol. For memory-less sources, ths s merely the entropy of Research artcle Fuzzy nformaton theory Zarand Indan Socety for Educaton and Envronment (See) Indan J.Sc.Technol. 023 each symbol, whle, n the case of a statonary stochastc process, t s r = n n lm H ( n n, n 2, 3,...) that s, the condtonal entropy of a symbol gven all the prevous symbols generated. For the more general case of a process that s not necessarly statonary, the average rate s r = lm H (, 2, 3, 4,...) n n that s, the lmt of the jont entropy per symbol. For statonary sources, these two expressons gve the same result. It s common n nformaton theory to speak of the "rate" or "entropy" of a language. Ths s approprate, for example, when the source of nformaton s Englsh prose. The rate of a source of nformaton s related to ts redundancy and how well t can be compressed, the subject of source codng (Arndt, 2004). Channel capacty Communcatons over a channel such as an ethernet wre s the prmary motvaton of nformaton theory. As anyone who's ever used a telephone (moble or landlne) knows, however, such channels often fal to produce exact reconstructon of a sgnal; nose, perods of slence, and other forms of sgnal corrupton often degrade qualty. How much nformaton can one hope to communcate over a nosy (or otherwse mperfec channel? Consder the communcatons process over a dscrete channel. A smple model of the process s shown below: Here represents the space of messages transmtted and Y the space of messages receved durng a unt tme over our channel. Let y x) be the condtonal probablty dstrbuton functon of Y gven. We wll consder y x) to be an nherent fxed property of our communcatons channel (representng the nature of the nose of our channel). Then the jont dstrbuton of and Y s completely determned by our channel and by our choce of f(x), the margnal dstrbuton of messages we choose to send over the channel. Under these constrants, we would lke to maxmze the rate of nformaton, or the sgnal, we can communcate over the channel. The approprate measure for ths s the mutual nformaton, and ths maxmum mutual nformaton s called the channel capacty and s gven by: C = max I( ; Y) f Ths capacty has the followng property related to communcatng at nformaton rate R (where R s usually bts per symbol). For any nformaton rate R < C and codng error ε > 0, for large enough N, there exsts a code of length N and rate R and a decodng algorthm, such that the maxmal probablty of block error s ε; that s, t

5 Indan Journal of Scence and Technology Vol. 3 No. 9 (Sep 200) ISSN: s always possble to transmt wth arbtrarly small block error. In addton, for any rate R > C, t s mpossble to transmt wth arbtrarly small block error. Channel codng s concerned wth fndng such nearly optmal codes that can be used to transmt data over a nosy channel wth a small codng error at a rate near the channel capacty. Capacty of partcular channel models A contnuous-tme analog communcatons channel subject to Gaussan nose see Shannon Hartley theorem. A bnary symmetrc channel (BSC) wth crossover probablty p s a bnary nput, bnary output channel that flps the nput bt wth probablty p. The BSC has a capacty of ( p) bts per channel use, where H b H b s the bnary entropy functon to the base 2 logarthm: no known attack can break them n a practcal amount of tme. Informaton theoretc securty refers to methods such as the one-tme pad that are not vulnerable to such brute force attacks. In such cases, the postve condtonal mutual nformaton between the plantext and cphertext (condtoned on the ke can ensure proper transmsson, whle the uncondtonal mutual nformaton between the plantext and cphertext remans zero, resultng n absolutely secure communcatons. In other words, an eavesdropper would not be able to mprove hs or her guess of the plantext by ganng knowledge of the cphertext but not of the key. However, as n any other cryptographc system, care must be used to correctly apply even nformaton-theoretcally secure methods; the Venona project was able to crack the one-tme pads of the Sovet Unon due to ther mproper reuse of key materal (Ash, 990, Cover & Joy, 2006). A bnary erasure channel (BEC) wth erasure probablty p s a bnary nput, ternary output channel. The possble channel outputs are 0,, and a thrd symbol 'e' called an erasure. The erasure represents complete loss of nformaton about an nput bt. The capacty of the BEC s - p bts per channel use. Applcatons to other felds Intellgence uses and secrecy applcatons: Informaton theoretc concepts apply to cryptography and cryptanalyss. Turng's nformaton unt, the ban, was used n the Ultra project, breakng the German Engma machne code and hastenng the end of WWII n Europe. Shannon hmself defned an mportant concept now called the uncty dstance. Based on the redundancy of the plantext, t attempts to gve a mnmum amount of cphertext necessary to ensure unque decpherablty. Informaton theory leads us to beleve t s much more dffcult to keep secrets than t mght frst appear. A brute force attack can break systems based on asymmetrc key algorthms or on most commonly used methods of symmetrc key algorthms (sometmes called secret key algorthms), such as block cphers. The securty of all such methods currently comes from the assumpton that Entropy by fuzzy probablty In ths secton we use the fuzzy probablty nstead of probablty n entropy. Defnton. Let Ω be a sample space and P be a probablty measure on Ω. If A ~ s a fuzzy event of Ω, then probablty of A ~ s defned as ~ A( w) d w) f Ω s not dscerete ~ Ω A) = A ~ ( w) w) f Ω s dscerete w Ω Example 2. Let tossng a con and A ~ be a small number occurs and B ~ be approxmately number 5. Then the events are: ~ A = {,,,, } ~ B = {,,,, } In ths case the probablty of event A ~ s ~ ~ p ( A) = A( Research artcle Fuzzy nformaton theory Zarand Indan Socety for Educaton and Envronment (See) Indan J.Sc.Technol. t Ω ~ = A () {}) + A ~ (2) {2}) + K + A ~ (6) {6}) = = 2.6 = and ~ p ( B ~ ) = B( = t Ω 6 Therefore we can n above example the probablty of a fuzzy event may be dffer wth the (crsp) probablty of an

6 Indan Journal of Scence and Technology Vol. 3 No. 9 (Sep 200) ISSN: event snce the fuzzy event was dffered wth (crsp) event. In entropy formula H ( p,..., p ) = C p log p we = consder the fuzzy probablty nstead of (crsp) probablty for generalzaton of ths noton and we ntroduce the fuzzy entropy. Therefore we can generalze all of the above notons and then we can ntroduce the fuzzy nformaton theory. Defnton 3. Let the entropy formula p be fuzzy probablty. Then we can use H ( p,..., p ) = C p log p = and compute the entropy for the fuzzy event wth fuzzy probablty nstead of (crsp) probablty. For convenence we put C = and then we have H ( p,..., p ) = p log p. = Example 4. Consder the fuzzy events of Example 2; we can compute the entropy for these fuzzy events. ~ ~ 2 H ( A, B) = p log p = (0.433log(0.433) log(0.46)) = = Concluson In ths paper we study the nformaton theory, entropy, codng and applcatons. We ntroduced the entropy wth fuzzy probablty as a generalzaton of entropy wth probablty whch s studed before. Snce we lve n fuzzy world and always we work wth vague notons therefore for transferrng the fuzzy events we must use another way snce studed method was about crsp events. We hope that ths noton s used for compactfcaton the data for transformaton and decrease the speed of networks. As a next work we can study ths noton and develop ths method for practcal problem. Acknowledgement The author would lke to express hs sncere thanks to the referees for ther valuable suggestons and comments. References. Anderson DR (2003) Some background on why people n the emprcal scences may want to better understand the nformaton-theoretc methods. en/coeevoluton/events/tms/why. Retreved on Dec. 30th, Arndt Chrstoph (2004) Informaton measures, nformaton and ts descrpton n scence and engneerng. Sgnals and Communcaton Technology, Sprnger Seres. 3. Ash B. Robert (990) Informaton theory. Interscence, NY; Dover, NY. 4. Burnham KP and Anderson DR (2002) odel selecton and multmodel nference: A practcal nformaton-theoretc approach, 2nd Edton, Sprnger Scence, NY. 5. Cover. Thomas and Joy A. Thomas (2006) Elements of nformaton theory. 2nd Edton, Wley- Inter-scence, NY. 6. Csszar Imre and Janos Korner (997) Informaton theory: codng theorems for dscrete memoryless systems. Akadema Kado, 2nd edton. 7. Davd J and C. ackay (2003) Informaton theory, nference, and learnng algorthms. Unversty Press, Cambrdge. 8. Gallager Robert (968) Informaton theory and relable communcaton. John Wley & Sons, NY. 9. Gbson Jerry D (998) Dgtal Compresson for ultmeda: Prncples and Standards. organ Kaufmann. g=pa56&dq=entropy-ate+condtonal &as_brr=3&e=ygdsrtzggkjupqka2l2xdw&sg=o0 UCtf0xZOflPIexPrjOKPgNc#PPA57,. 0. Goldman Stanford (2005) Informaton theory. Dover, NY.. Jaynes ET (957) Informaton theory and statstcal mechancs. Phys. Rev. 06, Kolmogorov Andrey (968) Three approaches to the quanttatve defnton of nformaton. Intl.J. Computer athematcs. Problems of Informaton Transmsson. No., ansurpur asud (987) Introducton to nformaton theory. Prentce Hall, NY. 4. Raymond W. Yeung (2008) Informaton theory and network codng. Sprnger. 5. Reza Fazlollah (994) An ntroducton to nformaton theory. Dover, NY. 6. Reke F, Warland D, Ruyter van Stevennck R and Balek Spkes W (997) Explorng the neural code. The IT press. 7. Shannon CE (948) A athematcal Theory of Communcaton. Bell System Techncal J. 27, & Shannon CE and Warren Weaver (949) The athematcal theory of communcaton. Unv of Illnos Press. 9. Yeung Raymond W (2002) A frst course n nformaton theory. Kluwer Academc/Plenum Publ. Research artcle Fuzzy nformaton theory Zarand Indan Socety for Educaton and Envronment (See) Indan J.Sc.Technol.