Channel Encoder. Channel. Figure 7.1: Communication system

Transcription

1 Chapter 7 Processes The model of a communcaton system that we have been developng s shown n Fgure 7.. Ths model s also useful for some computaton systems. The source s assumed to emt a stream of symbols. The channel may be a physcal channel between dfferent ponts n space, or t may be a memory whch stores nformaton for retreval at a later tme, or t may be a computaton n whch the nformaton s processed n some way. Source Encoder Compressor Channel Encoder Channel Input Output (Symbols) (Symbols) Channel Decoder Expander Source Decoder Fgure 7.: Communcaton system Fgure 7. shows the module nputs and outputs and how they are connected. A dagram lke ths s a useful overvew of the operaton of a system, but other representatons are also useful. In ths chapter we develop two abstract models that are general enough to represent each of these boxes n Fgure 7., but show the flow of nformaton quanttatvely. Because each of these boxes n Fgure 7. processes nformaton n some way, t s called a processor and what t does s called a process. The processes we consder here are Dscrete: The nputs are members of a set of mutually exclusve possbltes, only one of whch occurs at a tme, and the output s one of another dscrete set of mutually exclusve events. Fnte: The set of possble nputs s fnte n number, as s the set of possble outputs. Memoryless: The process acts on the nput at some tme and produces an output based on that nput, gnorng any pror nputs. Author: Paul Penfeld, Jr. Ths document: Verson.6, March, 200. Copyrght c 200 Massachusetts Insttute of Technology Start of notes back next 6.050J/2.0J home page Ste map Search About ths document Comments and nqures 82

2 7. Types of Process Dagrams 83 Nondetermnstc: The process may produce a dfferent output when presented wth the same nput a second tme (the model s also vald for determnstc processes). Because the process s nondetermnstc the output may contan random nose. Lossy: It may not be possble to see the nput from the output,.e., determne the nput by observng the output. Such processes are called lossy because knowledge about the nput s lost when the output s created (the model s also vald for lossless processes). 7. Types of Process Dagrams Dfferent dagrams of processes are useful for dfferent purposes. The four we use here are all recursve, meanng that a process may be represented n terms of other, more detaled processes of the same sort, nterconnected. Conversely, two or more connected processes may be represented by a sngle hgher-level process wth some of the detaled nformaton suppressed. The processes represented can be ether determnstc (noseless) or nondetermnstc (nosy), and ether lossless or lossy. Block Dagram: Fgure 7. (prevous page) s a block dagram. It shows how the processes are connected, but very lttle about how the processes acheve ther purposes, or how the connectons are made. It s useful for vewng the system at a hghly abstract level. An nterconnecton n a block dagram can represent many bts. Crcut Dagram: If the system s made of logc gates, a useful dagram s one showng such gates nterconnected. For example, Fgure 7.2 s an AND gate. Each nput and output represents a wre wth a sngle logc value, wth, for example, a hgh voltage representng and a low voltage 0. The number of possble bt patterns of a logc gate s greater than the number of physcal wres; each wre could have two possble voltages, so for n-nput gates there would be 2 n possble nput states. Often, but not always, the components n logc crcuts are determnstc. Probablty Dagram: A process wth n sngle-bt nputs and m sngle-bt outputs can be modeled by the probabltes relatng the 2 n possble nput bt patterns and the 2 m possble output patterns. For example, Fgure 7.3 (next page) shows a gate wth two nputs (four bt patterns) and one output. An example of such a gate s the AND gate, and ts probablty model s shown n Fgure 7.4. Probablty dagrams are dscussed further n Secton 7.2. Informaton Dagram: A dagram that shows explctly the nformaton flow between processes s useful. In order to handle processes wth nose or loss, the nformaton assocated wth them can be shown. Informaton dagrams are dscussed further n Secton 7.6. Fgure 7.2: Crcut dagram of an AND gate 7.2 Probablty Dagrams The probablty model of a process wth n nputs and m outputs, where n and m are ntegers, s shown n Fgure 7.5. The n nput states are mutually exclusve, as are the m output states. If ths process s mplemented by logc gates the nput would need at least log 2 (n) but not as many as n wres.

3 7.2 Probablty Dagrams Fgure 7.3: Probablty model of a two-nput one-output gate Fgure 7.4: Probablty model of an AND gate Ths model for processes s conceptually smple and general. It works well for processes wth a small number of bts. It was used for the bnary channel n Chapter 6. Unfortunately, the probablty model s awkward when the number of nput bts s moderate or large. The reason s that the nputs and outputs are represented n terms of mutually exclusve sets of events. If the events descrbe sgnals on, say, fve wres, each of whch can carry a hgh or low voltage sgnfyng a boolean or 0, there would be 32 (2 5 ) possble events. It s much easer to draw a logc gate, wth fve nputs representng physcal varables, than a probablty process wth 32 nput states. Ths exponental exploson of the number of possble nput states gets even more severe when the process represents the evoluton of the state of a physcal system wth a large number of atoms. For example, the number of molecules n a mole of gas s Avogadro s number N A = If each atom had just one assocated boolean varable, there would be 2 N A states, far greater than the number of partcles n the unverse. And there would not be tme to even lst all the partcles, much less do any calculatons: the number of mcroseconds snce the bg bang s less than Despte ths lmtaton, the probablty dagram model s useful conceptually. Let s revew the fundamental deas n communcatons, ntroduced n Chapter 6, n the context of such dagrams. We assume that each possble nput state of a process can lead to one or more output state. For each n nputs m outputs Fgure 7.5: Probablty model

4 7.2 Probablty Dagrams 85 nput denote the probablty that ths nput leads to the output j as c j. These transton probabltes c j can be thought of as a table, or matrx, wth as many columns as there are nput states, and as many rows as output states. We wll use as an ndex over the nput states and j over the output states, and denote the event assocated wth the selecton of nput as A and the event assocated wth output j as B j. The transton probabltes are propertes of the process, and do not depend on the nputs to the process. The transton probabltes le between 0 and, and for each ther sum over the output ndex j s, snce for each possble nput event exactly one output event happens. If the number of nput states s the same as the number of output states then c j s a square matrx; otherwse t has more columns than rows or vce versa. 0 c j (7.) = j c j (7.2) Ths descrpton has great generalty. It apples to a determnstc process (although t may not be the most convenent a truth table gvng the output for each of the nputs s usually smpler to thnk about). For such a process, each column of the c j matrx contans one element that s and all the other elements are 0. It also apples to a nondetermnstc channel (.e., one wth nose). It apples to the source encoder and decoder, to the compressor and expander, and to the channel encoder and decoder. It apples to logc gates and to devces whch perform arbtrary memoryless computaton (sometmes called combnatonal logc n dstncton to sequental logc whch can nvolve pror states). It even apples to transtons taken by a physcal system from one of ts states to the next. It apples f the number of output states s greater than the number of nput states (for example channel encoders) or less (for example channel decoders). If a process nput s determned by random events A wth probablty dstrbuton p(a ) then the varous other probabltes can be calculated. The condtonal output probabltes, condtoned on the nput, are The uncondtonal probablty of each output p(b j ) s p(b j A ) = c j (7.3) p(b j ) = c j p(a ) (7.4) Fnally, the jont probablty of each nput wth each output p(a, B j ) and the backward condtonal probabltes p(a B j ) can be found usng Bayes Theorem: 7.2. Example: AND Gate p(a, B j ) = p(b j )p(a B j ) (7.5) = p(a )p(b j A ) (7.6) = p(a )c j (7.7) The AND gate s determnstc (t has no nose) but s lossy, because knowledge of the output s not generally enough to nfer the nput. The transton matrx s [ ] [ ] c0(00) c 0(0) c 0(0) c 0() 0 = (7.8) c (00) c (0) c (0) c () The probablty model for ths gate s shown n Fgure 7.4.

5 7.2 Probablty Dagrams 86 =0 () j=0 =0 ε ε j=0 = () j= = ε 7 ε j= (a) No errors (b) Symmetrc errors Fgure 7.6: Probablty models for error-free bnary channel and symmetrc bnary channel Example: Bnary Channel The bnary channel s well descrbed by the probablty model. Its propertes, many of whch were dscussed n Chapter 6, are summarzed below. Consder frst a noseless bnary channel whch, when presented wth one of two possble nput values 0 or, transmts ths value fathfully to ts output. Ths s a very smple example of a dscrete memoryless process. We represent ths channel by a probablty model wth two nputs and two outputs. To ndcate the fact that the nput s replcated fathfully at the output, the nner workngs of the box are revealed, n Fgure 7.6(a), n the form of two paths, one from each nput to the correspondng output, and each labeled by the probablty (). The transton matrx for ths channel s [ ] [ ] c00 c 0 0 = (7.9) c 0 c 0 The nput nformaton I for ths process s bt f the two values are equally lkely, or f p(a 0 ) p(a ) the nput nformaton s ( ) ( ) I = p(a 0 ) log 2 + p(a ) log p(a 0 ) 2 (7.0) p(a ) The output nformaton J has a smlar formula, usng the output probabltes p(b 0 ) and p(b ). Snce the nput and output are the same n ths case, t s always possble to nfer the nput when the output has been observed. The amount of nformaton out, J, s the same as I, the amount n: J = I. Ths noseless channel s effectve for ts ntended purpose, whch s to permt the recever, at the output, to nfer the value at the nput. Next, let us suppose that ths channel occasonally makes errors. Thus f the nput s the output s not always, but wth the bt error probablty ε s flpped to the wrong value 0, and hence s correct only wth probablty ε. Smlarly, for the nput of 0, the probablty of error s ε. Then the transton matrx s [ ] [ ] c00 c 0 ε ε = (7.) c 0 c ε ε Ths model, wth random behavor, s sometmes called the Symmetrc Bnary Channel (SBC), symmetrc n the sense that the errors n the two drectons (from 0 to and vce versa) are equally lkely. The probablty dagram for ths channel s shown n Fgure 7.6(b), wth two paths leavng from each nput, and two paths convergng on each output. Clearly the errors n the SBC ntroduce some uncertanty nto the output over and above the uncertanty that s present n the nput sgnal. Intutvely, we can say that nose has been added, so that the output s composed n part of desred sgnal and n part of nose. Or we can say that some of our nformaton s lost n the channel. Both of these effects have happened, but as we wll see they are not always related; t s

6 7.3 Informaton, Loss, and Nose 87 possble for processes to ntroduce nose but have no loss, or vce versa. In Secton 7.3 we wll calculate the amount of nformaton lost or ganed because of nose or loss, n bts. Loss of nformaton happens because t s no longer possble to tell wth certanty what the nput sgnal s, when the output s observed. Loss shows up n drawngs lke Fgure 7.6(b) where two or more paths converge on the same output. Nose happens because the output s not determned precsely by the nput. Nose shows up n drawngs lke Fgure 7.6(b) where two or more paths dverge from the same nput. Despte nose and loss, however, some nformaton can be transmtted from the nput to the output (.e., observaton of the output can allow one to make some nferences about the nput). We now return to our model of a general dscrete memoryless nondetermnstc lossy process, and derve formulas for nose, loss, and nformaton transfer (whch wll be called mutual nformaton ). We wll then come back to the symmetrc bnary channel and nterpret these formulas. 7.3 Informaton, Loss, and Nose For the general dscrete memoryless process, useful measures of the amount of nformaton presented at the nput and the amount transmtted to the output can be defned. We suppose the process state s represented by random events A wth probablty dstrbuton p(a ). The nformaton at the nput I s the same as the entropy of ths source. (We have chosen to use the letter I for nput nformaton not because t stands for nput or nformaton but rather for the ndex that goes over the nput probablty dstrbuton. The output nformaton wll be denoted J for a smlar reason.) I = ( ) p(a ) log 2 (7.2) p(a ) Ths s the amount of uncertanty we have about the nput f we do not know what t s, or before t has been selected by the source. A smlar formula apples at the output. The output nformaton J can also be expressed n terms of the nput probablty dstrbuton and the channel transton matrx: J = j = j ( ) p(b j ) log 2 p(b j ) ( ) ( ) c j p(a ) log 2 c jp(a ) (7.3) Note that ths measure of nformaton at the output J refers to the dentty of the output state, not the nput state. It represents our uncertanty about the output state before we dscover what t s. If our objectve s to determne the nput, J s not what we want. Instead, we should ask about the uncertanty of our knowledge of the nput state. Ths can be expressed from the vantage pont of the output by askng about the uncertanty of the nput state gven one partcular output state, and then averagng over those states. Ths uncertanty, for each j, s gven by a formula lke those above but usng the reverse condtonal probabltes p(a B j ) ( ) p(a B j ) log 2 (7.4) p(a B j ) Then your average uncertanty about the nput after learnng the output s found by computng the average over the output probablty dstrbuton,.e., by multplyng by p(b j ) and summng over j

7 7.3 Informaton, Loss, and Nose 88 L = j = j p(b j ) ( ) p(a B j ) log 2 p(a B j ) ( ) p(a, B j ) log 2 p(a B j ) (7.5) Note that the second formula uses the jont probablty dstrbuton p(a, B j ). We have denoted ths average uncertanty by L and wll call t loss. Ths term s approprate because t s the amount of nformaton about the nput that cannot be found by examnng the output state; n ths sense t got lost n the transton from nput to output. In the specal case that the process allows the nput state to be dentfed unquely for all possble output states, the process s lossless and, as you would expect, L = 0. It was proved n Chapter 6 that L I or, n words, that the uncertanty after learnng the output s less than (or perhaps equal to) the uncertanty before. Ths result was proved usng the Gbbs nequalty. The amount of nformaton we learn about the nput state upon beng told the output state s our uncertanty before beng told, whch s I, less our uncertanty after beng told, whch s L. We have just shown that ths amount cannot be negatve, snce L I. As n Chapter 6, we denote the amount we have learned as M = I L, and call ths the mutual nformaton between nput and output. Ths s an mportant quantty because t s the amount of nformaton that gets through the process. To recaptulate the relatons among these nformaton quanttes: I = ( ) p(a ) log 2 (7.6) p(a ) L = p(b j ) ( ) p(a B j ) log 2 (7.7) p(a j B j ) M = I L (7.8) 0 M I (7.9) 0 L I (7.20) Processes wth outputs that can be produced by more than one nput have loss. These processes may also be nondetermnstc, n the sense that one nput state can lead to more than one output state. The symmetrc bnary channel wth loss s an example of a process that has loss and s also nondetermnstc. However, there are some processes that have loss but are determnstc. An example s the AND logc gate, whch has four mutually exclusve nputs and two outputs 0 and. Three of the four nputs lead to the output 0. Ths gate has loss but s perfectly determnstc because each nput state leads to exactly one output state. The fact that there s loss means that the AND gate s not reversble. There s a quantty smlar to L that characterzes a nondetermnstc process, whether or not t has loss. The output of a nondetermnstc process contans varatons that cannot be predcted from knowng the nput, that behave lke nose n audo systems. We wll defne the nose N of a process as the uncertanty n the output, gven the nput state, averaged over all nput states. It s very smlar to the defnton of loss, but wth the roles of nput and output reversed. Thus N = = p(a ) j p(a ) j ( ) p(b j A ) log 2 p(b j A ) ( ) c j log 2 c j (7.2)

8 7.4 Determnstc Examples 89 Steps smlar to those above for loss show analogous results. What may not be obvous, but can be proven easly, s that the mutual nformaton M plays exactly the same sort of role for nose as t does for loss. The formulas relatng nose to other nformaton measures are lke those for loss above, where the mutual nformaton M s the same: J = ( ) p(b j ) log 2 (7.22) p(b j ) N = p(a ) ( ) c j log 2 (7.23) c j j M = J N (7.24) 0 M J (7.25) It follows from these results that 0 N J (7.26) 7.3. Example: Symmetrc Bnary Channel J I = N L (7.27) For the SBC wth bt error probablty ε, these formulas can be evaluated, even f the two nput probabltes p(a 0 ) and p(a ) are not equal. If they happen to be equal (each 0.5), then the varous nformaton measures for the SBC n bts are partcularly smple: I = bt (7.28) L = N = ε log 2 ( ε M = ε log 2 ( ε J = bt (7.29) ) + ( ε) log 2 ( ) ε ) ) ( ε) log 2 ( ε (7.30) (7.3) The errors n the channel have destroyed some of the nformaton, n the sense that they have prevented an observer at the output from knowng wth certanty what the nput s. They have thereby permtted only the amount of nformaton M = I L to be passed through the channel to the output. 7.4 Determnstc Examples Ths probablty model apples to any system wth mutually exclusve nputs and outputs, whether or not the transtons are random. If all the transton probabltes c j are equal to ether 0 or, then the process s determnstc. A smple example of a determnstc process s the N OT gate, whch mplements Boolean negaton. If the nput s the output s 0 and vce versa. The nput and output nformaton are the same, I = J and there s no nose or loss: N = L = 0. The nformaton that gets through the gate s M = I. See Fgure 7.7(a). A slghtly more complex determnstc process s the exclusve or, XOR gate. Ths s a Boolean functon of two nput varables and therefore there are four possble nput values. When the gate s represented by

9 7.4 Determnstc Examples (a) NOT gate (b) XOR gate Fgure 7.7: Probablty models of determnstc gates a crcut dagram, there are two nput wres representng the two nputs. When the gate s represented as a dscrete process usng a probablty dagram lke Fgure 7.7(b), there are four mutually exclusve nputs and two mutually exclusve outputs. If the probabltes of the four nputs are each 0.25, then I = 2 bts, and the two output probabltes are each 0.5 so J = bt. There s therefore bt of loss, and the mutual nformaton s bt. The loss arses from the fact that two dfferent nputs produce the same output; for example f the output s observed the nput could be ether 0 or 0. There s no nose ntroduced nto the output because each of the transton parameters s ether 0 or,.e., there are no nputs wth multple transton paths comng from them. Other, more complex logc functons can be represented n smlar ways. However, for logc functons wth n physcal nputs, a probablty dagram s awkward f n s larger than 3 or 4 because the number of nputs s 2 n Error Correctng Example The Hammng Code encoder and decoder can be represented as dscrete processes n ths form. Consder the (3,, 3) code, otherwse known as trple redundancy. The encoder has one -bt nput (2 values) and a 3-bt output (8 values). The nput s wred drectly to the output and the nput 0 to the output 000. The other sx outputs are not connected, and therefore occur wth probablty 0. See Fgure 7.8(a). The encoder has N = 0, L = 0, and M = I = J. Note that the output nformaton s not three bts even though three physcal bts are used to represent t, because of the ntentonal redundancy. The output of the trple redundancy encoder s ntended to be passed through a channel wth the possblty of a sngle bt error n each block of 3 bts. Ths nosy channel can be modelled as a nondetermnstc process wth 8 nputs and 8 outputs, Fgure 7.8(b). Each of the 8 nputs s connected wth a (presumably) hgh-probablty connecton to the correspondng output, and wth low probablty connectons to the three other values separated by Hammng dstance. For example, the nput 000 s connected only to the outputs 000 (wth hgh probablty) and 00, 00, and 00 each wth low probablty. Ths channel ntroduces nose snce there are multple paths comng from each nput. In general, when drven wth arbtrary bt patterns, there s also loss. However, when drven from the encoder of Fgure 7.8(a), the loss s 0 bts because only two of the eght bt patterns have nonzero probablty. The nput nformaton to the nosy channel s bt and the output nformaton s greater than bt because of the added nose. Ths example demonstrates that the value of both nose and loss depend on both the physcs of the channel and the probabltes of the nput sgnal. The decoder, used to recover the sgnal orgnally put nto the encoder, s shown n Fgure 7.8(c). The transton parameters are straghtforward each nput s connected to only one output. The decoder has loss (snce multple paths converge on each of the ouputs) but no nose (snce each nput goes to only one output).

10 7.5 Capacty (a) Encoder (b) Channel (c) Decoder Fgure 7.8: Trple redundancy error correcton 7.5 Capacty In Chapter 6 of these notes, the channel capacty was defned. Ths concept can be generalzed to other processes. Call W the maxmum rate at whch the nput state of the process can be detected at the output. Then the rate at whch nformaton flows through the process can be as large as W M. However, ths product depends on the nput probablty dstrbuton p(a ) and hence s not a property of the process tself, but on how t s used. A better defnton of process capacty s found by lookng at how M can vary wth dfferent nput probablty dstrbutons. Select the largest mutual nformaton for any nput probablty dstrbuton, and call that M max. Then the process capacty C s defned as C = W M max (7.32) It s easy to see that M max cannot be arbtrarly large, snce M I and I log 2 n where n s the number of dstnct nput states. In the example of symmetrc bnary channels, t s not dffcult to show that the probablty dstrbuton that maxmzes M s the one wth equal probablty for each of the two nput states. 7.6 Informaton Dagrams An nformaton dagram s a representaton of one or more processes explctly showng the amount of nformaton passng among them. It s a useful way of representng the nput, output, and mutual nformaton and the nose and loss. Informaton dagrams are at a hgh level of abstracton and do not dsplay the detaled probabltes that gve rse to these nformaton measures. It has been shown that all fve nformaton measures, I, J, L, N, and M are nonnegatve. It s not necessary that L and N be the same, although they are for the symmetrc bnary channel whose nputs have equal probabltes for 0 and. It s possble to have processes wth loss but no nose (e.g., the XOR gate), or nose but no loss (e.g., the nosy channel for trple redundancy). It s convenent to thnk of nformaton as a physcal quantty that s transmtted through ths process much the way physcal materal may be processed n a producton lne. The materal beng produced comes nto the manufacturng area, and some s lost due to errors or other causes, some contamnaton may be added (lke nose) and the output quantty s the nput quantty, less the loss, plus the nose. The useful product s the nput mnus the loss, or alternatvely the output mnus the nose. The flow of nformaton through a dscrete memoryless process s shown usng ths paradgm n Fgure 7.9. An nterestng queston arses. Probabltes depend on your current state of knowledge, and one observer s knowledge may be dfferent from another s. Ths means that the loss, the nose, and the nformaton transmtted are all observer-dependent. Is t OK that mportant engneerng quanttes lke nose and loss depend on who you are and what you know? If you happen to know somethng about the nput that

11 7.7 Cascaded Processes 92 N nose I nput M J output L loss Fgure 7.9: Informaton flow n a dscrete memoryless process your colleague does not, s t OK for your desgn of a nondetermnstc process to be dfferent, and to take advantage of your knowledge? Ths queston s somethng to thnk about; there are tmes when your knowledge, f correct, can be very valuable n smplfyng desgns, but there are other tmes when t s prudent to desgn usng some worst-case assumpton of nput probabltes so that n case the nput does not conform to your assumed probabltes your desgn stll works. Informaton dagrams are not often used for communcaton systems. There s usually no need to account for the nose sources or what happens to the lost nformaton. However, such dagrams are useful n domans where nose and loss cannot occur. One example s reversble computng, a style of computaton n whch the entre process can, n prncple, be run backwards. Another example s quantum communcatons, where nformaton cannot be dscarded wthout affectng the envronment Notaton Dfferent authors use dfferent notaton for the quanttes we have here called I, J, L, N, and M. In hs orgnal paper Shannon called the nput probablty dstrbuton x and the output dstrbuton y. The nput nformaton I was denoted H(x) and the output nformaton J was H(y). The loss L (whch Shannon called equvocaton ) was denoted H y (x) and the nose N was denoted H x (y). The mutual nformaton M was denoted R. Shannon used the word entropy to refer to nformaton, and most authors have followed hs lead. Frequently nformaton quanttes are denoted by I, H, or S, often as functons of probablty dstrbutons, or ensembles. In physcs entropy s often denoted S. Another common notaton s to use A to stand for the nput probablty dstrbuton, or ensemble, and B to stand for the output probablty dstrbuton. Then I s denoted I(A), J s I(B), L s I(A B), N s I(B A), and M s I(A; B). If there s a need for the nformaton assocated wth A and B jontly (as opposed to condtonally) t can be denoted I(A, B) or I(AB). 7.7 Cascaded Processes Consder two processes n cascade. Ths term refers to havng the output from one process serve as the nput to another process. Then the two cascaded processes can be modeled as one larger process, f the nternal states are hdden. We have seen that dscrete memoryless processes are characterzed by values of I, J, L, N, and M. Fgure 7.0(a) shows a cascaded par of processes, each characterzed by ts own parameters. Of course the parameters of the second process depend on the nput probabltes t encounters, whch are determned by the transton probabltes (and nput probabltes) of the frst process. But the cascade of the two processes s tself a dscrete memoryless process and therefore should have ts own fve parameters, as suggested n Fgure 7.0(b). The parameters of the overall model can be calculated

12 7.7 Cascaded Processes 93 N N N 2 I J =I 2 J 2 I J M M 2 M L L 2 L (a) The two processes (b) Equvalent sngle process Fgure 7.0: Cascade of two dscrete memoryless processes ether of two ways. Frst, the transton probabltes of the overall process can be found from the transton probabltes of the two models that are connected together; n fact the matrx of transton probabltes s merely the matrx product of the two transton probablty matrces for process and process 2. All the parameters can be calculated from ths matrx and the nput probabltes. The other approach s to seek formulas for I, J, L, N, and M of the overall process n terms of the correspondng quanttes for the component processes. Ths s trval for the nput and output quanttes: I = I and J = J 2. However, t s more dffcult for L and N. Even though L and N cannot generally be found exactly from L, L 2, N and N 2, t s possble to fnd upper and lower bounds for them. These are useful n provdng nsght nto the operaton of the cascade. It can be easly shown that snce I = I, J = I 2, and J = J 2, L N = (L + L 2 ) (N + N 2 ) (7.33) It s then straghtforward (though perhaps tedous) to show that the loss L for the overall process s not always equal to the sum of the losses for the two components L + L 2, but nstead so that the loss s bounded from above and below. Also, so that f the frst process s nose-free then L s exactly L + L 2. There are smlar formulas for N n terms of N + N 2 : 0 L L L + L 2 (7.34) L + L 2 N L L + L 2 (7.35) 0 N 2 N N + N 2 (7.36) N + N 2 L 2 N N + N 2 (7.37) Smlar formulas for the mutual nformaton of the cascade M follow from these results: M L 2 M M I (7.38) M L 2 M M + N L 2 (7.39) M 2 N M M 2 J (7.40) M 2 N M M 2 + L 2 N (7.4)

13 7.7 Cascaded Processes 94 Other formulas for M are easly derved from Equaton 7.9 appled to the frst process and the cascade, and Equaton 7.24 appled to the second process and the cascade: M = M + L L = M + N + N 2 N L 2 = M 2 + N 2 N = M 2 + L 2 + L L N (7.42) where the second formula n each case comes from the use of Equaton Note that M cannot exceed ether M or M 2,.e., M M and M M 2. Ths s consstent wth the nterpretaton of M as the nformaton that gets through nformaton that gets through the cascade must be able to get through the frst process and also through the second process. As a specal case, f the second process s lossless, L 2 = 0 and then M = M. In that case, the second process does not lower the mutual nformaton below that of the frst process. Smlarly f the frst process s noseless, then N = 0 and M = M 2. The channel capacty C of the cascade s, smlarly, no greater than ether the channel capacty of the frst process or that of the second process: C C and C C 2. Other results relatng the channel capactes are not a trval consequence of the formulas above because C s by defnton the maxmum M over all possble nput probablty dstrbutons the dstrbuton that maxmzes M may not lead to the probablty dstrbuton for the nput of the second process that maxmzes M 2.