Universal communication part II: channels with memory

Transcription

1 Unversal councaton part II: channels wth eory Yuval Lontz, Mer Feder Tel Avv Unversty, Dept. of EE-Systes Eal: arxv: v2 [cs.it] 20 Mar 203 Abstract Consder councaton over a channel whose probablstc odel s copletely unnown vector-wse and s not assued to be statonary. Councaton over such channels s challengng because nowng the past does not ndcate anythng about the future. The exstence of relable feedbac and coon randoness s assued. In a prevous paper t was shown that the Shannon capacty cannot be attaned, n general, f the channel s not nown. An alternatve noton of capacty was defned, as the axu rate of relable councaton by any bloc-codng syste used over consecutve blocs. Ths rate was shown to be achevable for the odulo-addtve channel wth an ndvdual, unnown nose sequence, and not achevable for soe channels wth eory. In ths paper ths capacty s shown to be achevable for general channel odels possbly ncludng eory, as long as ths eory fades wth te. In other words, there exsts a syste wth feedbac and coon randoness that, wthout nowledge of the channel, asyptotcally perfors as well as any bloc code, whch ay be desgned nowng the channel. For non-fadng eory channels a weaer type of capacty s shown to be achevable. Index Ters Unnown channels, Unversal councaton, Feedbac councaton, Arbtrarly varyng channels, Channels wth eory. I. INTRODUCTION Consder councaton over a channel whch has a general probablstc structure. In other words, the nfnte length output Y depends on the nfnte length nput X through an arbtrary vector-wse probablty functon P Y X Y n X, n =, 2,..., whch s unnown to the transtter and the recever. Partcular cases of such a channel nclude any unnown functonal relaton between the nput and output sequences, as well as arbtrarly varyng channels, copound channels [] and channels wth an ndvdual state sequence [2][3][4][5]. In the current paper, an attept s ade to eep the odel as general as possble,.e. nze any assuptons on P Y X, except for causalty. Wthout feedbac, councaton over such a channel s lted, as the councaton rate, and the codeboo would have to be selected n advance. Therefore, the exstence of a relable feedbac ln s assued. Two tradtonal odels, whch relate to partcular cases of the current proble, are the arbtrarly varyng channel AVC odel [] and the copound fnte state channel copound- FSC odel [6]. In the AVC odel, the channel s assued to be controlled by a sequence of states whch s arbtrary and unnown to the transtter and the recever. In the copound channel odel, the channel s assued to be arbtrarly selected fro a faly of possble channels. In both odels, the capacty s the axu rate of relable councaton that can be guaranteed. Both odels do not gve a satsfyng answer to the current proble: the fundaental reason s that these odels focus on capacty,.e. before nowng the channel, one s requred to fnd a rate of relable transsson whch can be guaranteed a-pror. Clearly, f the channel s copletely general, the copound/avc capacty s zero, as t s possble, for exaple, that a channel wth zero capacty wll be selected. In both odels entoned, constrants on the faly of channels, or on the possble state sequences need to be defned, and these constrants do not see sutable for natural channels. In addton to ths fundaental gap, the odels consdered under the AVC and copound-fsc fraewors are qute lted, n a way that does not see to capture the possble coplexty of an unnown natural channel. For exaple, ost papers on AVC consder only eoryless channels, and the copound-fsc s statonary. Usng feedbac, the councaton rate can be adapted, so that one does not have to cot to a councaton rate a-pror. Several wors by us and other authors consdered the gans fro such adaptaton [2][3][7][5]. The frst queston to as s, how the target councaton rate should be defned? The sought rate RP Y X can be a functon of the channel, but should be unversally attanable wthout pror nowledge of the channel, and should have an operatonal eanng. Put sply, one would le to have a unversal ode whch can be connected over any channel, and would attan rates, whch, ay not be optal, but would at least be justfable and wll not ae one regret for not odelng the channel and usng a ode optzed for the channel. In a prevous paper [4], the proble of deternng such a councaton rate was addressed. In general, the Shannon capacty [8] of the channel, CP Y X s not attanable unversally wth feedbac, when the channel s unnown. Ths s exeplfed n [4] through the sple exaple of the oduloaddtve channel wth an unnown nose sequence, where the Shannon capacty of each channel ndvdually s postve the logarth of the alphabet sze, whle the axu relable councaton rate that can be guaranteed a-pror s zero. The proble of deternng a unversally-achevable rate s slar to the source codng proble of settng a copresson rate for an ndvdual sequence. As n the unversal source codng proble, due to the rchness of the odel faly, there s a large gap between the perforance that can be attaned unversally and the perforance that can be attaned wthout constrants, when nowng the specfc odel the

2 2 Shannon capacty and ths gap requres ltng the abltes of the reference syste. Followng the sprt of the fnte state copressblty of Lepel and Zv [9], we proposed to set as a target, the best rate that can be relably attaned by a syste eployng fnte bloc encodng successvely over the nfnte channel. The supreu of these rates s tered the Iteratve-Fnte-Bloc IFB capacty and denoted C IFB P Y X. When the channel s statonary and ergodc, then the IFB capacty equals the Shannon capacty. Ths otvates consderng the IFB capacty as a goal. It s easy to see that the IFB capacty s not unversally achevable for copletely general odels. The counter exaple n [4] s of a faly consstng of only two bnary channels, tered password channels, where the frst nput bt X deternes whether the channel becoes good or bad for eternty, and where the values of X atchng each state are opposte n the two channels. There s no way for the unversal syste to correctly guess X wth hgh probablty. The concluson s that the IFB capacty s not unversally attanable for soe channels wth nfnte eory. On the other hand, the IFB capacty was shown to be asyptotcally attanable for the class of odulo-addtve channels wth an ndvdual, unnown nose sequence. In ths case, t was further shown, that the IFB capacty s related to the fnte state copressblty of the nose sequence, and the schee attanng t uses the Lepel-Zv source encoder [9] to generate decodng etrcs. The result n [4] reles crucally on two propertes of the odulo addtve channel: The channel s eoryless wth respect to the nput x.e. current behavor s not affected by prevous values of the nput. 2 The capacty achevng nput dstrbuton s fxed unfor..d. regardless of the nose sequence. To avod these assuptons t s requred to address the eory of the channel and the settng of the councaton pror. The second ltaton, rases the queston, how the nput dstrbuton should be adapted, f the channel changes arbtrarly over te? Ths queston was the center of [5], where unversal predcton ethods were used to set the councaton pror. The focus of that paper s on channels whch are eoryless n the nput, and therefore can be defned by an unnown sequence of eoryless channels P Y X Y n X n = n = W Y X. It s shown there that the capacty of the te-averaged channel W y x = n n = W y x can be unversally attaned usng feedbac and coon randoness wthout nowng {W, and that ths value s the axu rate that can be acheved unversally and does not depend on the order of the channels n the sequence. The noton of unversalty used n [5] s dfferent and weaer than the IFB unversalty, snce the rate s only copared wth other rates that could have been unversally attaned. In the current paper, deas fro [4] and [5] are cobned to generalze the prevous results. It s shown that the IFB capacty s asyptotcally unversally attanable for any channel wth a fadng eory,.e. where the effect of the channel hstory on the far future s vanshng. In ths sense, the two assuptons used n the prevous paper [4] are avoded as uch as possble, and the assuptons ade on the channel are sgnfcantly nzed. The fadng eory condton ncludes as partcular cases eoryless arbtrarly varyng channels as well as copound ndecoposable fnte state channels [0]. Here, an exaple s gven of a class of fnte state channels where the state s a non-hoogenous Marov chan, whch satsfy the fadng eory condton. Consderng channels where eory of the past s not necessarly fadng, t ay stll be possble to councate unversally over the channel, f t s not alcously desgned le the password channel descrbed above. The advantage of the IFB reference class whch enables t to wn over any unversal syste s ts ablty to deterne such a codeboo that wll not only enable relable transsson, but wll also eep the channel n a favorable state, whereas the unversal syste does not now the long ter effects of certan nput sybols or dstrbutons. An alternatve forulaton s proposed, where the reference syste s crppled, so that t cannot enjoy the ablty to shape the past: the encoder and decoder operate over fnte blocs, however the error probablty s requred to be sall n the worst case channel state hstory pror to each bloc, and average over blocs. Ths odels a stuaton where the reference encoder and decoder are thrown each te nto a dfferent locaton n te, where the past state ght have been arbtrary. It s not requred to have good perforance n each of these events, but only on average. Ths alternatve reference syste s tered arbtrary-fnte-bloc AFB and the sae unversal syste s shown to asyptotcally approach the respectve AFB capacty, wthout requrng that the channel eory s fadng. Ths reference class s less natural than the IFB, yet t enables releasng constrants on the channel. Note that there are several alternatve defntons of a lted reference class for the unversal councaton proble [4]. Most notably, Msra and Wessan [] generalzed the an results of [4] to fnte-state councaton systes wth feedbac. For the sae of splcty the current paper focuses on the basc odel of reference systes usng bloc codng. Although the current result s purely theoretcal, t supples otvaton for usng copettve unversalty n councaton. II. PROBLEM SETTING, DEFINITIONS AND MAIN RESULT The defntons n Sectons II-A-II-D repeat and extend the respectve defntons n the prevous paper [4]. Secton II-E foralzes the an result of the paper. A. Notaton Vectors are denoted by boldface letters. Sub-vectors are defned by superscrpts and subscrpts: x j [x j, x j+,..., x ]. x j equals the epty strng f < j. The subscrpt s soetes reoved when t equals,.e. x x. For a vector x, x [] x + + denotes the -th bloc of length n the vector. For brevty, vectors wth slar ranges are soetes joned together, for exaple, the notaton xy s used nstead of x y. Exponents and logs are base 2. Rando varables are dstngushed fro ther saple values by captal letters. Z + denotes the set of non-negatve ntegers.

3 3 X Y n-l - Part on whch probablty s evaluated n 2 - Condton s allowed to affect probablty - Condtonng wealy affects probablty Fg.. An llustraton of the fadng eory condton Defnton 2. IQ, W denotes the utual nforaton obtaned when usng a pror Q over a channel W,.e. t s the utual nforaton IQ, W = IX; Y between two rando varables wth the jont probablty X, Y = QX W Y X. CW denotes the channel capacty CW = ax Q IQ, W. B. Channel odel Let x and y be nfnte sequences denotng the nput and the output respectvely, where each letter s chosen fro the alphabets X, Y respectvely, x X, y Y. Throughout the current paper the nput and output alphabets are assued to be fnte. A channel P Y X s defned through the probablstc relatons P Y X y n x = Y n = y n X = x for n =, 2,.... A fnte length output sequence s consdered n order to ae the probablty well defned. Soetes, ths probablty wll be nforally referred to as Y X, and should be understood as the sequence of these dstrbutons for n =, 2,.... Defnton. The channel defned by Y n X s tered causal f for all n: n Y n X = Y n X n. All the defntons below ncludng IFB/AFB capacty pertan to causal channels. Ths characterzaton of a causal channel s slar to the defnton used by Han and Verdú [8] and references theren. Ths defnton s also lted n assung the channel starts fro a nown state at te 0. However ths does not lt the current settng, because an arbtrary ntal state can be odeled by consderng the faly of channels wth all possble ntal states. Note that non causalty that conssts of bounded negatve delays can always be copensated by applyng a delay to the output. Defnton 2. The channel s tered a fadng eory channel f for any h > 0 there exsts L and a sequence of causal condtonal vector dstrbuton functons {P n, such that for all n and n: Y n X, Y n L P n Y n X n L h, 2 where the L nor s calculated over Y n, and defned by gy y gy The dfference between the ters on the LHS of 2 s that P n does not nclude XY n L see Fg., and thus the fadng eory condton asserts that the dependence of the condtonal dstrbuton of future outputs, on the channel state at the far past, decays. Notce that the condtonal dstrbuton Yn X, Y n L s copletely defned by the channel, snce t s condtoned on the entre nput X. On the other hand, the condtonal dstrbuton Yn X n L ay depend also on the nput dstrbuton through the unspecfed sybols X n L. Therefore, the dstrbuton P n n Defnton 2 s not dentcal to Yn X n L. On the other hand, oposton shows that Yn X n L obtaned wth any nput dstrbuton yelds a legtate P n. The fadng eory condton does not ply statonarty or ergodcty. The eoryless arbtrary varyng channel odel consdered n [5] s fadng eory, and so are the FSC [0, 4.6] or copound-fsc odels [6], f the underlyng FSC s ndecoposable. An exaple of a non-hoogeneous fnte state channel wth fadng eory s presented n Secton V. C. IFB and AFB capacty The followng defntons lead to the defntons of IFB capacty and AFB capacty. Defnton 3 Reference encoder and decoder. A fnte length encoder E wth bloc length and a rate R s a appng E : {,..., M X fro a set of M expr essages to a set of nput sequences X. A respectve fnte length decoder D s a appng D : Y {,..., M fro the set of output sequences to the set of essages. Defnton 4 IFB error probablty. The average error probablty n teratve appng of the length encoder E and decoder D to b blocs over the channel P Y X s defned as follows: b essages,..., b are chosen as..d. unforly dstrbuted rando varables U{,..., M, =,..., b. The channel nput s set to X [] = E, =,..., b, and the decoded essage s ˆ = DY [] where Y s the channel output. The teratve appng s llustrated n Fg.2. The average error probablty s P e = b b = ˆ. Defnton 5 AFB error probablty. The average error probablty n arbtrary appng of the length encoder E and decoder D to b blocs over the channel P Y X s defned as P e = b b = P e. P e s the worst case per-bloc error probablty, defned as: [ P e = ax XY { ] 3 X [] = E, XY, DY [] where U{,..., M. Defnton 6 IFB/AFB achevablty. A rate R s teratedfnte-bloc IFB / arbtrary-fnte-bloc AFB achevable resp. over the channel P Y X, f for any ɛ > 0 there exst, b > 0 such that for any b > b there exst an encoder E and a decoder D wth bloc length and rate R for whch the average error probablty n teratve/arbtrary appng resp. of E, D to b blocs s at ost ɛ.

4 4 Ths s equvalent to statng that the l sup of the average error probablty wth respect to b s at ost ɛ. Defnton 7 IFB/AFB capacty. The IFB/AFB capacty of the channel P Y X s the supreu of the set of IFB/AFB achevable rates, and s denoted C IFB /C AFB resp.. By defnton, the AFB error probablty s at least as large as the IFB error probablty, and as a result, the AFB capacty s saller than, or equal to the IFB capacty. D. Copettve Unversalty In the followng, the propertes of the adaptve syste wth feedbac, and IFB/AFB-unversalty are defned. A randozed rate-adaptve transtter and recever for bloc length n wth feedbac are defned as follows see also foral defntons n [2, 5]: the transtter s presented wth a essage expressed by an nfnte bt sequence, and followng the recepton of n sybols, the decoder announces the acheved rate R, and decodes the frst nr bts. An error eans any of these bts dffers fro the bts of the orgnal essage sequence. Both encoder and decoder have access to a rando varable S the coon randoness dstrbuted over a chosen alphabet, and a causal feedbac ln allows the transtted sybols to depend on prevously sent feedbac fro the recever. The syste s llustrated n Fg. 3. The followng defnton states forally the noton of IFB/AFB-unversalty for rate adaptve systes: Defnton 8 IFB/AFB unversalty. Wth respect to a set of channels {P Y X, θ θ Θ not necessarly fnte or countable, a rate-adaptve councaton syste possbly usng feedbac and coon randoness s called IFB/AFB unversal f for every channel n the faly and any ɛ, δ > 0 there s n large enough such that when the syste s operated over n channel uses, then wth probablty ɛ, the essage s correctly decoded and the rate s at least C IFB P Y X δ or C AFB P Y X δ resp.. Notce that the defntons above and specfcally Defntons 6,8 do not requre unfor convergence wth respect to the channel,.e. the nuber of channels uses n or blocs b for whch the requreents hold ay be a functon of the channel. E. The an result Theore. For any ɛ > 0 there exsts a sequence of adaptve rate systes over a bloc of sze N wth feedbac and coon randoness, for growng values of N, such that wth a probablty of at least ɛ the essage s receved correctly wth a rate of: R UNI [N] ax [C IFB δn IFB, C AFB δn AFB ], 4 where δn AFB 0 for any causal channel, and δn IFB 0 N N for any causal fadng eory channel. Furtherore, ths can be attaned wth any postve rate of the feedbac ln. Ths ples that the syste s IFB unversal over the set of causal fadng eory channels, and AFB unversal over the set of causal channels, accordng to Defnton 8. Whle the syste does not depend on the channel, the convergence rate of δn IFB, δafb N does. III. COMMUNICATION SCHEME AND PROOF OUTLINE A. The councaton schee In [5], a councaton schee for adaptng the pror over an arbtrarly varyng channel whch s eoryless n the nput was descrbed. Cobnng Theore 3 and Lea 9 of [5] yelds: Lea. [Lea 9 of [5]] For every ɛ, δ > 0 there exsts n and a constant c, such that for any n n there s an adaptve rate syste wth feedbac and coon randoness, such that for any channel Y n X n : The probablty of error s at ost ɛ 2 The rate satsfes R CW SUBJ C wth probablty at least δ where and W SUBJ = n n Y = y X = x, X, Y, 5 = ln 2 n 4 C = c. 6 n The unversal councaton schee for attanng the clas of Theore s as follows. The nfnte te s dvded nto epochs of ncreasng length, nubered =, 2,.... In the frst epoch, the schee of Lea descrbed n [5] s operated over N sybols. In the second epoch, the channel nputs and outputs are joned nto pars,.e. super-sybols of denson 2, and the schee s operated over N 2 such supersybols. In epoch, the schee s operated over N supersybols of denson 2 Fg.4. Snce all N are fnte, the denson of the super-sybols used grows ndefntely wth te. The paraeters of the schee are chosen as follows. Let ɛ > 0 the chosen error probablty. Choose any C > 0, and let ɛ = 2 ɛ 2. The length of the -th epoch, N, s chosen such that: It s equal to or larger than the value of n gven by Lea for the paraeters ɛ = δ = ɛ. 2 The value of C gven by Lea for n = N s not larger than C the chosen value. 3 If the end of the next epoch N + would occur beyond sybol N, then the current epoch N s extended to reach sybol N. The second requreent aes sure that there s no ore than a constant loss fro capacty per epoch, whle the denson of the super-sybol of each epoch s growng, and therefore the loss per sybol tends to zero. The values of δ and ɛ chosen per epoch, guarantee that the overall probablty of error s not larger than = ɛ = 2ɛ and slarly the overall probablty that at any epoch the rate falls below the rate declared n the lea s at ost 2ɛ. Ths way the overall probablty of havng an error or fallng below the guaranteed

5 5 w w 2 w 3 w 4 w 5 Encoder Encoder Encoder Encoder Encoder t = Channel t = 50 Decoder Decoder Decoder Decoder Decoder ŵ ŵ 2 ŵ 3 ŵ 4 ŵ 5 Fg. 2. An llustraton of teratve appng used for the defnton of average error probablty see Defnton 4. The sae encoder and decoder are used over each of the b = 5 blocs of = 0 channel uses, and the average error probablty s coputed. N Fg. 4. Epoch N 2 N 3 N 4 super-sybols Epoch 2 Epoch 3 Epoch 4 q = 2 = 8 Dvson nto epochs and super-sybols n the unversal schee rate s at ost ɛ. Note that the epoch duratons N are fxed and do not depend on the essage or receved sgnal. The schee does not need to now the IFB/AFB bloc length, rate and error probablty, and the exact relaton between L, h gven by the fadng eory condton Defnton 2. Its only paraeters are the nput and output alphabets, the nuber of sybols N, and the error probablty ɛ. The cla of Theore, that any postve feedbac rate s suffcent, sply follows fro the fact [5] that ths s true for the schee of Lea. The schee of Lea s a fnte horzon schee,.e. n has to be set n advance, and there s no guarantee on the rate at the ddle of an epoch. Due to ths techncal ltaton, the unversal schee proposed here s also of a fnte horzon,.e. the sybol N n whch the syste s perforance s to be easured s specfed n advance. It s clear fro the constructon of the schee that ths ltaton s techncal and nor. B. oof outlne Followng s the outlne of the proof. The value Y = y X = x, X, Y appearng n the defnton of W SUBJ 5 s the probablty of a certan output sybol to appear gven a certan nput sybol at te, where the hstory of the channel XY attans the specfc value that occurred durng the unversal syste s operaton. Y = y X = x, X, Y s a rando varable and depends both on the channel and on the unversal councaton syste behavor. As a result, the rate W SUBJ guaranteed by Lea s also a rando varable and depends on the jont nput-output dstrbuton nduced by the unversal councaton schee. Ths rate s tered subjectve snce t would be dfferent had a dfferent schee operated on the sae channel. The baselne for coparson wth the reference syste s the pessstc average channel capacty, obtaned by replacng the hstory XY by an arbtrary state, and tang the worst-case state sequence worst case hstory,.e. the one that yelds the nu capacty. The rate attaned by the unversal syste for a partcular state sequence would be at least as large. For super-sybols, the averaged channel relates to the jont dstrbuton over the super-sybol, where the state XY q refers to the nput and output sequences before the start of the super-sybol. The unversal syste s shown to asyptotcally attan a rate whch s at least the weghted average of the pessstc average channel capactes easured over the epochs oposton 2. Next, the reference syste wth bloc sze s copared to the unversal syste durng epoch, where the super-sybol length s q = 2. Consder a set of super-sybols n hops of l +j : l Z +, j =,...,. Snce the nuber of sybols between the start of two successve super-sybols n each of these algnent sets dvdes by, n each of these supersybols, the reference syste s blocs and the super-sybols algn,.e. the IFB/AFB blocs begn at the sae locaton wth respect to the begnnng of the super-sybol see Fg.5. Therefore, there s an equvalence between the average error probablty of the reference syste over these supersybols, and the error probablty that would be attaned for the collapsed channel, generated by randoly and unforly drawng one of the super-sybols n the set and operatng the reference syste over ths channel. Due to ths equvalence, the reference syste s rate, for a gven average error probablty, s lted by the capacty of the collapsed channel. For the IFB case, ths collapsed channel s nduced not only by the channel law, but also by the behavor of the reference syste n prevous blocs. When replacng the

6 6 w essage Encoder x X Channel P Y X y x y Y f F = {0, feedbac Decoder R rate ŵ essage S coon randoness S Fg. 3. Rate adaptve encoder-decoder par wth feedbac, over an unnown channel collapsed channel wth a slar channel, where the hstory XY q before each super-sybol s forced to a specfc value, then due to the fadng eory assupton, fro soe pont n the bloc onward, the two channels becoe slar n L sense. Due to ths slarty, the ncrease n error probablty, when exchangng the orgnal collapsed channel wth the new one, s sall Lea 3. The new channel s not subjectve,.e. t s only a functon of the channel P Y X and not of the syste operatng over t. For the AFB case, ths transton s not needed, as the desred relaton stes edately fro the defnton. Usng a varant of Fano s nequalty, the rate of the IFB/AFB syste s related to the capacty of the pessstc average channel easured over each of the algnent sets of supersybols 34. The pessstc average channel over the epoch, s the average of the average channels easured over the algnent sets. Averagng channels ay nduce a loss of at ost log n capacty Lea 4. Ths results n a bound on the pessstc average channel durng each epoch, as a functon of the IFB/AFB capacty, and the IFB/AFB error probablty durng the epoch. Note that at ths stage, the error probablty of the reference syste cannot be dsssed as beng sall, t s guaranteed to be sall only on average, over growng ntervals n te. Tang the weghted average of the pessstc capactes over the epochs enables relatng the rate of the unversal syste to the rate and the average error probablty of the reference syste, where the latter tends to zero. All overheads, such as the ones related to algnent of the blocs to the super-sybols, the te t taes the channel eory to fade, the log penalty for xng channels, vansh asyptotcally as the super-sybol length ncreases ndefntely wth te. Although the result of ths paper s sple, the proof s far fro elegant, and the syste, although sple, s not effcent n convergng to ts target. Let us hope that a ore drect proof wll be found n the future. IV. PROOF OF THE MAIN RESULT A. Addtonal notaton for the proof Addtonal notaton requred for the proof s defned below. The proof copares a stuaton where the reference IFB syste operates on the channel to the unversal syste operatng on the sae channel. Although the channels are the sae, the jont dstrbuton of the nput and the output s dfferent due to the dfferent encoders. The channel nput and outputs when the unversal syste operates are denoted by X, Y, whle X, Ỹ denote the channel nputs and outputs when the reference syste operates. Snce both systes operate on the sae channel the condtonal dstrbuton s the sae,.e. Y n = y X = x = Ỹn = y X = x The followng sybols have constant eanng throughout the proof. denotes the epoch ndex, and q denotes the denson of the super-sybol, whch s a functon of q = 2. denotes the bloc length of the reference syste. N denotes the overall nuber of sybols and M denotes the overall nuber of epochs. B. Channel odel prelnares The followng sple conclusons follow fro the defntons of the causal and the fadng eory channel. Regardng Defnton of a causal channel, note that the sae holds for argnal dstrbutons of Y n e.g. the dstrbuton of Y n as s easly shown by suaton over. Another consequence of Defnton s that for n, n 2, n 3, n 4 n, the followng condtonal dstrbuton can also be gven as a functon of a fnte nput: Yn n2 Yn n4 3, X = Yn2 n Yn n4 3 X Yn n4 3 X = Yn2 n Yn n4 3 X n Yn n4 3 X n = Yn n2 Yn n4 3, X n. Two sple consequences of Defnton 2 fadng eory channel are gven below. The proof s sple and deferred to Appendx A. oposton. For a causal fadng eory channel, the followng holds If 2 holds for a certan, then t holds for any saller as long as n. Ths ples that 2 only needs to be establshed for large enough. 2 For any nput dstrbuton and for any > n, Yn X, Y n L Yn X n L 2h. 8 In other words, the property apples when P n s replaced wth the true probablty Yn X n L, obtaned wth any nput dstrbuton. C. A guarantee on the pessstc rate The rate W SUBJ s subjectve n the sense that t depends on the jont nput-output dstrbuton nduced by the unversal councaton schee, and would be dfferent had a dfferent schee operated on the sae channel. 7

7 7 In the followng, a lower rate s defned, but such that s a oposton 2. Assue that for each epoch wth supersybol length q = 2, the pessstc capacty satsfes: functon of the channel alone. Let W [q] SUBJ denote the subjectve average channel over n super-sybols of denson q: W [q] SUBJy q x q = n Y [q] = y X [q] = x, XY q. q C[q,] PMA C δ, 3 n = 9 where δ 0. Let C M N = 2 N C denote the Ths channel s tered subjectve snce t depends on the specfc nput-output dstrbuton nduced by the unversal schee for the unversal schee of Secton III-A, over N sybols average of C weghted by the relatve epoch duratons. Then, when operatng on the channel whch s dfferent, n general, and M epochs, wth probablty at least ɛ, the essage s fro the jont dstrbuton nduced by a reference syste. correctly decoded and the rate satsfes: Furtherore, snce Y [q] = y X [q] = x, XY q s a rando varable dependng on the hstory XY q, also R UNI [N] C δ N, 4 W [q] SUBJ s a rando varable, whose dstrbuton depends on the jont dstrbuton nduced by the schee. Also note that where δ N 0. N whle the condtonng on XY q represent what truly oof: By ts constructon and Lea, n epoch, wth happened as a rando varable, ths channel s not an probablty at least ɛ, the schee attans the followng eprcal channel, snce the probablty above represents rate, per super-sybol: what would have happened, hypothetcally at the output, f one forced the nput X [q] = x. Let S [q] denote the state before super-sybol, S [q] XY q. As the channel s not a fnte state channel, the = alphabet sze of S [q] ncreases wth. Consder the average channel when the hstory X q, Y q obtans a specfc value s : W [q] y q x q ; {s n = = n Y [q] = y X [q] = x, S [q] = s. n = 0 In other words, for fxed nput and output, ths s the average probablty to see the specfc output gven the specfc nput when the channel had been n a specfc state. Ths s no longer a rando varable, but a functon of {s. The pessstc average channel capacty s defned as the worst capacty of W [q] ; {s n = for any state sequence. C [q] PMA = nf C {s n = W [q] y x; {s n =. Note that n tang the nu, does not requre that the state sequence satsfes the natural constrant gven by the recurson S = S, X, Y,.e. t s allowed to nclude socalled contradctons. By defnton, ths rate lower bounds the capacty of the subjectve averaged channel: C W [q] SUBJy q x q = C W [q] y x; {s n = s=s nf C W [q] y x; {s n = {s n = = C [q] PMA. 2 Snce n each epoch, the unversal schee asyptotcally attans the capacty of W [q] SUBJ easured over the epoch, t also attans C PMA. [q] The next proposton antans that f t s guaranteed that the noralzed pessstc capacty, s asyptotcally on average above soe rate C then the schee wll asyptotcally approach the rate C. The pessstc capacty wth super-sybol q easured over epoch s denoted C [q,] PMA. R C W [q,] SUBJ y q x q 2 C C [q,] PMA C qc δ. 5 The nuber of bts sent durng ths epoch s at least R N. Let M denote the nuber of epochs untl te N, where N = M = 2 N. Wth probablty at least ɛ recall: ɛ = 2 = ɛ, there s no decodng error and the rate up to te N s at least: M = R UNI [N] R N N 5 M 2 C δ C N N = = C N = C δ M. M δ C 2 N = 6 where δ M 0. The last step stes fro the followng M sple lea see Appendx C: Lea 2. For a postve, onotonc n non-decreasng sequence a n and 0 δ n 0, = aδ n 0. Furtherore, the n = a n convergence s unfor over the values of {a n. Note that because the last epoch stretches to te N, the coeffcents a = 2 N vary as N s ncreased. However, accordng to the lea, t only atters that they rean onotonc and that the nuber of coeffcents grows wth N. The next subsectons relate C [q,] PMA to the rate obtaned by the reference syste for a certan error probablty. The proof for the IFB and AFB cases s qute slar. For the purpose of clarty, the proof below focuses on the ore coplex IFB case, and at the end, the odfcatons requred for the AFB case are dscussed.

8 8 Algnent set B Algnent set B 2 Algnent set B 3 L frst sybols of the super-sybol L Super-sybol =q sybols Sybols L to q = = 2 = 3 L Fg. 6. At worst, + 2 blocs ay fully or partally overlap wth the frst L sybols of any super-sybol. = 4 = 5 = 6 Te n B blocs Collapsed channel ɛ j Average error probablty over bloc of subset B j Fg. 5. The algnent of reference syste blocs n the unversal syste s super-sybols. The large dar rectangles are the supersybols of length q, wth the trangles denotng the frst L sybols. The lght rectangles are the reference syste blocs of length where here = 3. There are three algnent sets B j, j =, 2, 3. In the exaple, n B j = 3, 3, 4 for j =, 2, 3. The blocs that are not accounted for n n B j are ared wth an x. The error probablty ɛ j refers to the sae reference syste bloc over dfferent super-sybols n the algnent set. The collapsed channel s averaged across an algnent set. D. IFB syste perforance durng a sngle epoch Consder the reference syste coposed of an encoder and a decoder operatng over bloc sze, and the unversal syste n epoch, wth super-sybol length q = 2. For splcty, as long as a sngle epoch s concerned, the sybols and super-sybols of the epoch are denoted by ndces startng fro.e. =, 2,..., N q or =, 2,..., N respectvely. In the followng, the propertes of the IFB syste such as rate and error probablty are lned to a channel averaged over super-sybols. Frst, let us consder the channel fro the IFB syste s pont of vew. X, Ỹ denote the nput and output vectors durng the epoch, where the jont dstrbuton depends on the jont behavor of the IFB encoder and the channel. Consder the set of super-sybols wth ndex B j { = l + j : l Z +, N for j =,...,,.e. the set of super-sybols n hops of the reference bloc sze startng fro the j-th super-sybol. B j are not necessarly of the sae sze. In each of the super-sybols n a set B j, the reference syste s blocs begn at the sae locaton wth respect to the begnnng of the super-sybol see Fg.5. The sets B j are tered algnent sets. To use the fadng eory assupton, the reference syste perforance s consdered only over sybols L through q out of the q sybols n the super-sybol. In each subset B j, consder the blocs whch copletely overlap wth sybols L through q. The nuber of such blocs per super-sybol n the set B j s denoted n B j. The nuber of sybols n epoch whch are not ncluded n any of these blocs for any B j s denoted n 0, where by the above defntons: n 0 = N q j B j n B j, 7 =.e. n 0 equals the total nuber of sybols n the epoch, nus the nuber of sybols covered per super-sybol, sued over the subsets. n 0 can be bounded fro above by consderng that no ore than L + 2 blocs ay fully or partally overlap wth the frst L sybols of any supersybol see Fg.6, and therefore at ost L + 2 sybols per super-sybol are lost, hence n 0 L + 2 N. 8 Ths calculaton accounts correctly for the specal cases of the frst and the last super-sybols n the epoch, as one ay wrap the epoch around ts tal, and agne that the end of the epoch s cyclcally connected to ts begnnng. It s convenent to noralze n 0 and the nuber of sybols n each set B j by the total nuber of sybols n the epoch, and loo at the relatve szes: where by 7: λ j B j n B j, j =,..., 9 N q n 0 8 λ 0 L + 2, 20 N q q j=0 λ j 7 =. 2 Consderng a specfc algnent set B j, because the reference syste s operaton s fxed durng these blocs, ts average error probablty can be related to the utual nforaton X [q] of the averaged collapsed channel. Denote by, Ỹ[q] the channel nput and output of the reference syste durng the -th super-sybol, and by Ỹ[q] q L the output durng sybols L to q of the super-sybol. Let X c,j, Ỹc,j denote a rando varable generated by a unfor selecton over B j of, n other words, X [q], Ỹ[q] q L X c,j, Ỹc,j [q] = X U, Ỹ[q] U q L, U UB j. 22 The jont dstrbuton of X c,j, Ỹc,j s: X c,j = x, Ỹc,j = y = B j B j { X[q] = x, Ỹ[q] q L = y. 23

9 9 Because the reference syste nduces the sae nput dstrbuton n all the super-sybols n B j,.e. = x s { X[q] constant for all B j, the argnal dstrbuton of X c,j equals the per-bloc dstrbuton for all B j X c,j = x = { { X[q] = x = X[q] = x, B j B j and hence the condtonal dstrbuton s: Ỹc,j = y X c,j = x = X c,j = x, Ỹc,j = y X c,j = x { = X[q] = x, Ỹ[q] U q L { = y B j B j X[q] = x = { B j Ỹ[q] q L = y X[q] = x. B j Denote the reference syste rate by R IFB. Consder eployng the reference syste over a super-sybol selected randoly and unforly over B j, where the essage s encoded n the n B j blocs whch are contaned n sybols L through q of the super-sybol. Denote the average error probablty whch s attaned for the -th bloc averaged over the channel and over all super-sybols n B j by ɛ j. The average error probablty over the n B j blocs s denoted ɛ j n B j nb j = ɛ j. It s convenent to defne the average error rate of the IFB syste n the epoch, ɛ IFB as the su of error probabltes over all blocs that begn n the epoch, noralzed by the approxate nuber of blocs n the epoch N q. Ths error probablty can be bounded as: nb j ɛ j= = B j ɛ j IFB N q = N q = B j n B j ɛ j j= λ j ɛ j, j= 26 where the nequalty s because the suaton on the rght sde only accounts for the error probablty over the blocs that are fully contaned wthn sybols L to q of any supersybol. E. Operaton of the IFB syste over a odfed channel The rando varables X c,j, Ỹc,j are nduced by the channel and the behavor of the reference syste. Not only s the dstrbuton of X c,j deterned by the codeboo dstrbuton of the reference encoder, but the channel behavor deternng Ỹ c,j s potentally affected by the nput dstrbuton nduced by the reference encoder on all prevous sybols. To account for the fadng eory of the channel, and relate these varables to the ones seen by the unversal syste, let us consder alternatve rando varable, representng an alternatve, specfc, channel state at the begnnng of the super-sybol. For a gven state sequence s, consder the rando varable Ỹs,j whch depends on X c,j through the followng condtonal dstrbuton: Ỹs,j = y X c,j = x = { B j Ỹ[q] B j q L = y X [q] = x, XỸ q = s 27 Because the current nuberng refers only to epoch, the notaton XỸ q forally refers to the nput and output of the channel wth the reference syste durng the epoch. However the eanng of XỸ q should be understood as the nput and output of the channel fro the begnnng of te potentally before epoch. Also, consderng that the reference encoder ay not be able to et all possble nput sequences, the eanng of condtonng on X should be understood as f the encoder was dsconnected and an nput value was forced nto the channel. Because the probablty on the RHS of 27 s condtoned on the entre past of X, and the channel s causal, ths probablty does not depend on the reference syste, but only on the channel. Therefore, the sae probablty would be attaned for the rando varables X, Y representng the nputs and outputs of the channel when the unversal syste s appled: Ỹs,j = y X c,j = x = { B j B j Y [q] q L = y X[q] = x, XY q = s 28 Usng the fadng eory assupton, t s shown below, that the error probablty obtaned when applyng the n B j blocs of the reference syste to the channel defned by 27 s not sgnfcantly worse than ts perforance over the channel of 25. Let xy { P = { Ỹ[q] Ỹ[q] q L q L = y X [q] = x = y X [q] = x, XỸ q = s. 29 Because the channel s assued to be causal and fadng eory, usng oposton, for any h > 0 there exsts L such that: xy P 2h, 30 and therefore by the trangle nequalty, the dfference between the two channels s bounded by: Ỹs,j = y X c,j = x Ỹc,j = y X c,j = x 25,27 = xy P B j B j xy P B j 2h. B j 3

10 0 Fro the L bound on the dfference between the condtonal probabltes, a bound on the ncrease n the error probablty s easly derved as follows: Lea 3. Let the error probablty of a gven encoder and decoder over the vector channel W y x be ɛ for =, 2. If for all X, W Y X W 2 Y X h W then ɛ ɛ 2 h W oof: Denote by E the event of error. Then, ɛ = E W = X,Y E XY X W Y X, 32 where the probablty of error gven XY does not depend on the channel and for a deternstc encoder and decoder t s ether 0 or dependng on whether Y belongs to the decson regon of X. As a result, ɛ ɛ 2 = X E XY W Y X W 2 Y X X Y X X = X X Y X Y E XY W Y X W 2 Y X W Y X W 2 Y X X W Y X W 2 Y X X h W X = h W. 33 Note that the lea apples to any event whose probablty s fxed as functon of X, Y. In the followng, t wll be appled to the event of an error n each of the blocs separately whle the channel s the channel over the super-sybol defned n 27. F. The average capacty per algnent set A lower bound on the capacty of the channel {Ỹs,j X c,j s obtaned by usng the fact the IFB syste delvers a certan rate wth a sall bloc error probablty. Followng s a varaton of Fano s nequalty, whch taes nto account that the errors are bloc errors rather than full essage errors. Denote by E the ndcator assocated wth the event of error n the -th bloc out of n B j blocs, and ɛ j = E[E ] the probablty of error on ths bloc over all super-sybols n B j, when the reference decoder s appled to the channel output Ỹ s,j 27. By applyng Lea 3 to the event of an error n the -th bloc, ɛ j ɛ j + 2h s obtaned where ɛ j s the error probablty of the sae bloc under the orgnal channel 25. The average error probablty over the blocs s denoted ɛ j nb j n B j = ɛ j. Whenever E = 0, then gven the channel output, R IFB bts of the nput becoe nown, whereas when E = these bts are unnown and have entropy at ost R IFB. Denote by the transtted essage a sequence of K j = n B j R IFB bts and by ˆ the decoded essage. The dervaton below uses the fact condtonng reduces entropy, and the concavty of the bnary entropy functon h b : H ˆ H, {E ˆ = H {E, ˆ + H{E ˆ {e H : E = e, ˆ{E = e + H{E e R IFB {E = e + {e = E [E ] R IFB + h b ɛ j = ɛ j R IFB + h b ɛ j n B jɛ j R IFB + n B jh b ɛ j = K j ɛ j + n B jh b ɛ j. HE 34 Usng the nforaton processng nequalty, the capacty of the channel s lower bounded as follows: C {Ỹs,j X c,j I X c,j ; Ỹs,j I; ˆ = H H ˆ K j K j ɛ j n B jh b ɛ j. 35 Defne h b p h b n p, 2 hb p as the onotone contnuaton of h b. h b p s non decreasng and concave. Then usng ɛ j ɛ j + 2h: C {Ỹs,j X c,j K j K j ɛ j n B jh b ɛ j K j ɛ j 2h n B jh b ɛ j + 2h. G. A bound on the pessstc averaged channel 36 To connect the capacty above to the pessstc averaged channel, the bound on the capacty of each average channel over a set B j needs to be lned wth the capacty over the averaged channel over the sets. For ths purpose the followng sple lea s used: Lea 4. Let W be a set of channels, and p 0, p = a probablty dstrbuton over the channels. Then p C W Hp C p W p C W. 37 The rght nequalty s based on convexty of the utual nforaton wth respect to the channel and the left nequalty s based on the fact the dfference between nowng and not nowng the ndex at the channel output s at ost the entropy of ths nforaton. The sple proof s deferred to Appendx B.

11 The averaged channel over the epoch wth a specfc state sequence s: W [q] y q x q ; s = N N = { Y [q] = y X [q] = x, XY q = s 38 Ths channel s capacty s at least as large of the capacty of the next channel, where the frst L outputs are reoved: W [q]\l y q L+ x q ; s = N { Y [q] q L N = y X[q] = x, XY q = s = N 28 = j= = j= B j { Y [q] B j {Ỹs,j = y N X c,j = x. The lea ples that C W [q] ; s C W [q]\l ; s = C Lea 4 j= q L = y X[q] = x, XY q = s B j {Ỹs,j = y N X c,j = x B j C {Ỹs,j = y N X c,j = x j= {{ H { Bj N j= C avg The last ter s upper bounded by H { Bj N. j= log, 4 and the frst ter C avg s bounded by 35 and substtutng K j = n B j R IFB : C avg 35 = j= j= 9 = qr IFB 2,26 B j K j ɛ j 2h n B jh b N ɛ j + 2h B j n B j R IFB ɛ j 2h h b N ɛ j + 2h λ j ɛ j 2h j= q λ 0 j= λ j λ 0 h b ɛ j + 2h qr IFB λ 0 ɛ IFB 2h q λ 0 h λ j b ɛ j + 2h λ 0 j= qr IFB λ 0 ɛ IFB 2h q h b λ 0 ɛ IFB + 2h. 42 Cobnng 4,42, dvdng by q and tang nfu over s yelds: q C[q,] PMA = nf C W [q] ; s s R IFB R IFB λ 0 + ɛ IFB + 2h h b λ 0 ɛ IFB + 2h log. q 43 For any L, λ 0 can be ade arbtrarly sall by tang large enough equvalently q large enough. Tang large enough such that λ 0 2. Thus, for large enough: q C[q,] PMA R IFB R IFB λ 0 + ɛ Alternatvely, for all : h b q C[q,] PMA R IFB,RIFB where 2ɛ IFB + 2h 2ɛ IFB,RIFB t = R IFB t + h b IFB + 2h log. q h,RIFB 2, 45 t, 46 and 2 s defned as the reander,.e. R IFB nus the RHS of 43 nus, and by 44, for large enough,,rifb 2 L R IFB λ 0 + log R L + 2 IFB + log. q q q 47,RIFB t s concave n t, tends to zero wth t 0 and decreases wth.,rifb 2 L tends to zero wth.

12 2 H. Concluson of the proof for the IFB case Now, ultple epochs are consdered, and oposton 2 s appled to bound the rate of the unversal schee. Suppose that over the N sybols and M epochs of the syste s operaton, the IFB syste acheves rate R IFB wth an average error probablty ɛ IFB. The defnton of ɛ IFB above 26 results n ɛ IFB = M 2 N N blocs = ɛ IFB where N blocs, the nuber of IFB blocs that begn n any sybol of the syste s operaton s upper bounded by N blocs N and so M R IFB,RIFB δ N N = = R IFB,RIFB 2ɛ IFB + 2h δ N = R IFB,RIFB 4ɛ IFB δ N, 2 N 2ɛ IFB + 2h 49 where the nequalty s due to the concavty of,rifb t. For an arbtrarly sall δ C, choose R IFB = C IFB δ C. By the defnton of the IFB capacty, there s a large enough and N large enough so that ɛ IFB can be ade arbtrarly sall. Therefore,RIFB 4ɛ IFB can be ade arbtrarly sall note that t decreases wth whle δ N 0. Therefore N for large enough N, the RHS of 49 can be ade arbtrarly close to C IFB. Ths proves the IFB unversalty of the proposed unversal syste. I. Modfcatons for the AFB case The proof for the AFB case s slar and the requred odfcatons are dscussed below. The sae defntons of Secton IV-D are used for the algnent sets up to Equaton 2, except L s set to L =. The defnton of X c,j, Ỹc,j s not needed, as the perforance of the AFB syste s drectly related to the constraned-state channel whose output s Ỹs,j. For the arbtrary sequence of states s, defne the channel Ỹs,j = y X c,j = x accordng to 28. Ths channel ples that the channel hstory s forced to s at the begnnng of each super-sybol. In each algnent set B j the n B j blocs of the AFB syste are apped to ths averaged channel. Clearly, the error probablty of the AFB syste when the state s forced to soe value at the begnnng of the super-sybol, s not worse than the error probablty n arbtrary appng defned n Defnton 5, where the state s forced to ts worst-case value just before the relevant bloc. Forally, let E l denote an ndcator of the event of error n the l-th bloc of the -th supersybol, a bloc whch begns at sybol n l of the supersybol, then when appng to the channel where only the ntal state s forced, the error probablty s: ] XY [E q l = s ɛ IFB 2 N ɛ IFB. 48 [ [ ] XY ] N XY = = E E E q =s 50, q l XY n l = s q+ Choose h = ɛ IFB and deterne the respectve L to satsfy where the terated expectaton law s appled. The nternal the fadng eory property note that ths choce s for the expectaton s by defnton upper bounded by P e l, the error purpose of analyss, and the schee tself s not aware of probablty n arbtrary appng Defnton 5 over the sae these values. Applyng oposton 2 wth C = R IFB,RIFB 2ɛ IFB + 2h and δ =,RIFB bloc, and therefore the error probablty wth the current 2 L, there exsts appng s upper bounded by the error probablty n arbtrary δ N 0 such that N appng. Denote as before by ɛ j the average error probablty over R UNI [N] C δ N the -th blocs n the B j algnent set, when the AFB = R IFB M syste s apped to the channel 2 N,RIFB 2ɛ Ỹs,j = y X c,j = x, IFB + 2h N and the average error probablty over the n B j blocs by = δ ɛ j nb j n B j = ɛ j, 26 now holds wth respect to the N average error n arbtrary appng over the epoch, where M E now the nequalty stes not only fro the fact that not all errors are accounted for, but n addton because the error probabltes ɛ j, ɛ j are upper bounded by the respectve errors obtaned by arbtrary appng. The transton to a odfed channel Secton IV-E s not requred n ths case and the proof s contnued wth the value h = 0. The rest of the proof proceeds as before Sectons IV-F,IV-G,IV-H, where ɛ AFB and R AFB replace ɛ IFB and R IFB, except that n 49 h s chosen to be zero rather than equal ɛ IFB. Ths concludes the proof of Theore. V. AN EXAMPLE OF A FADING MEMORY CHANNEL In the defnton of a fadng eory channel Defnton 2, the overall probablty of Y over the nfnte future fro n to s requred to be close n L sense to a dstrbuton that does not depend on the past. Ths rases the queston how strct s the requreent and whether t s satsfed by broad faly of channels. Below, an exaple s gven of a faly of fnte state channels wth non-hoogeneous transton probabltes that, under the assupton that there s a non-zero probablty to arrve fro any state to any state, satsfes the fadng eory requreent. Consder a fnte state channel where the state at each oent n te S belongs to the fnte set S. The probablty of each output letter s gven as a te-varyng functon of the nput letter and the current state Y X ; S = W Y X ; S, and the state sequence s a non-hoogeneous Marov chan whch depends on the nput va S S, X = T S S ; X. The jont probablty s therefore n {Y n S n X n, S 0 = T S S ; X W Y X ; S. = 5

13 3 If the Marov chan deternng the state transtons s such that, eventually, t s possble to ove fro any state to any state, then t s tered a ndecoposable Marov chan. Slarly, for constant state transton and channel probabltes T, W, Gallager [0, 4.6] defned the resultng fnte state channel as ndecoposable f the eory of the ntal state fades wth te Eq there. Here, for splcty, a strcter condton s assued: that t s possble to ove fro any state to any state wthn one step and wth a certan, nonvanshng probablty β > 0,.e. that S ; X : T S S ; X β. 52 It appears that ths condton can be relaxed and the results can be generalzed to ndecoposable Marov chans, under the assupton that there s soe nu probablty to arrve fro any state to any state wth a fnte nuber of steps, by sply treatng a bloc of sybols as a new super-sybol. However for splcty let us focus on ths type of channels, whch s also qute general. oposton 3. Any channel wth the structure defned above s a causal fadng eory channel. Specfcally, the L dstance n Defnton 2 s h 2 S β L+,.e. fades exponentally wth L. The rest of ths secton s devoted to the proof of ths proposton. The transton probablty ay be wrtten alternatvely as follows: re T S S ; X = λ + λt S S ; X, S 53 where λ = S β. Due to the condton 52, the reander s non negatve, and by sung both sdes of 53 over S t s easly seen that T re s a legtate probablty dstrbuton. Ths otvates the followng forulaton: consder a sequence of..d. Bernully rando varables A Berλ, whch are drawn ndependently of X and of prevous S -s. The next state s deterned as follows. If A = 0 then the next state s deterned by T re S S ; X. Otherwse, T re t s selected unforly wth equal probabltes. Ths results n the sae condtonal probablty T S S ; X due to 53. The fadng eory property stes fro the observaton that whenever A =, the eory of the past dsappears, and that over a long enough nterval, the probablty for such an event approaches one. Due to the ndependence of A n the sequence X and the prevous states, t s also ndependent of the past of Y, Hence Yn X, Y n L = Yn A n n L X n LXY n L A n n L = A n n L Y n X n LXY n L A n n L A n n L. 54 The dstrbuton condtoned on A n n L s: Yn X n LXY n L A n n L = Y n S n L S n X n LXY n L A n n L = S n L+,S n S n L X n LXY n L A n n L S n X n LXY n L A n n LS n L Yn X n, S n S n L+,S n S n L XY n L S n X n n LS n L A n n L. 55 Focusng on the last ter, t can be shown that t has a wea dependence on S n L. Gven {A, the sequence S reans a Marov chan, therefore for any n L < n S n S n L X n n L, A n n L = S S n L X n n L, A n n L 56 S S n S X n n L, A n n L. If A = then the frst ter s constant and ndependent of S n L and therefore S n S n L X n n L, n L An does not depend on S n L. The sae s trvally true for = n. The probablty that none of A n n L would be s λl+. Whenever any of A n n L s, because the last ter n 55 s ndependent of S n L, the su n 55 breas nto two ndependent sus and Yn X n L XYn L An L n does not depend on XY n L. Therefore, consderng the suaton n 54, t can be wrtten as: Yn X, Y n L = λ L+ P Y n X n L 57 + λ L+ P 2 Y n X, Y n L, where the probabltes P, P 2 are generated by splttng the su 54 to the sngle coponent that depends on X n L, Y n L and the other coponents that do not, and noralzng each part. Fro 57 the L dstance can be bounded: h = Yn X, Y n L P Y n X n L = λ L+ P Y n X n L P2 Y n X, Y n L λ L+ P Y n Xn L + P2 Y n X, Y n L = 2 λ L+. VI. DISCUSSION A. Coparson wth extng results 58 Table I copares the current results wth prevous and new results of us and other authors.