ε-capacity and Strong Converse for Channels with General State

Transcription

1 ε-capacity ad Strog Coverse for Chaels with Geeral State Marco Tomamichel Cetre for Quatum Techologies, Natioal Uiversity of Sigapore Vicet Y. F. Ta Istitute for Ifocomm Research, A*STAR & Dept. of ECE, Natioal Uiversity of Sigapore Abstract We cosider state-depedet memoryless chaels with geeral state available at both ecoder ad decoder. We establish the ε-capacity ad the optimistic ε-capacity. This allows us to prove a ecessary ad sufficiet coditio for the strog coverse to hold. We also provide a simpler sufficiet coditio o the first- ad secod-order statistics of the state process that esures that the strog coverse holds. I. INTRODUCTION We study the problem of chael codig with states where the chael, viewed as a stochastic kerel from the set of iputs ad states to the output, is discrete, memoryless ad statioary (time-ivariat), while the state is allowed to be a geeral source i the sese of Verdú-Ha 2, 3. The state is assumed to be kow ocausally at both ecoder ad decoder. This models the sceario i which the chael is o-ergodic or havig memory, where both o-ergodicity ad memory are iduced by the geeral state sequece. We derive the ε-capacity ad the optimistic ε-capacity 4 uder cost costraits by usig the iformatio spectrum method 2, 3 ad a tight coverse boud established by us 5. These capacities oly deped o the cumulative distributio fuctio (cdf) of the Cèsaro mea of the capacity-cost fuctios. This corroborates our ituitio because the chael is well-behaved, thus it does ot require characterizatio usig iformatio spectrum quatities ad, i particular, probabilistic limit operatios 2, 3. The oly complicatio that ca arise is due to the geerality of the state ad for this, we require probabilistic limits. Thus, we observe a decouplig of the radomess iduced by the chael ad the state. Usig the form of the ε-capacity ad its optimistic, we provide a ecessary ad sufficiet coditio for the chael to have the strog coverse 2, Sec. V 3, Def By usig Chebyshev s iequality, we provide a simpler sufficiet coditio for the strog coverse to hold. This coditio is based o first- ad secod-order statistics of the state process ad hece, is easier to verify. Fially, we provide examples to illustrate the various coditios. For more extesive results o this work icludig secod-order codig rates, we refer the reader to 6. A. Basic Defiitios II. PRELIMINARIES AND DEFINITIONS We assume throughout that X, Y ad S are fiite sets. Let P(X ) be the set of probability distributios o X. We also deote the set of chaels from X to Y as P(Y X ) = P(Y) X. I the followig, we let W P(Y X S) be a chael where X deotes the iput alphabet, S deotes the state alphabet ad Y deotes the output alphabet. The set of all x X that are admissible for the chael i state s S is B s (Γ) := x X b s (x) Γ for some fuctios b s : X R + ad Γ > 0. We do ot explicitly metio Γ if there are o cost costraits, i.e., if Γ =. Furthermore, we assume that the chael state S is a radom variable with probability distributio P S P(S). For ay P P(X S), we defie the coditioal distributio P W P(Y S) as P W (y s) = x P (x s)w (y x, s). The followig coditioal log-likelihood ratios are also of iterest: i(x; y s) := log W (y x, s) P W (y s), j W (y x, s) Q(x; y s) := log Q(y s), where the latter defiitio applies for ay Q P(Y S) with Q( s) W ( x, s) for every (x, s) X S. We deote the coditioal mutual iformatio as I(P, W P S ) := Ei(X; Y S) where (S, X, Y ) P S (s)p (x s)w (y x, s). Furthermore, the capacity-cost fuctio of the chael W s := W (, s) is defied as C s (Γ) := max I(P, W s). P P(B s(γ)) A code for the chael W with cost costrait Γ is defied by C := M, e, d where M is the message set, e : M S X is the ecoder ad d : Y S M is the decoder. The ecoder must satisfy e(m, s) B s (Γ) for all s S ad m M. For S P S, the average (for uiform M) ad maximum error probabilities are respectively defied as p avg (C; W, P S ) := PrM M ad p max (C; W, P S ) := max m M PrM M M = m, We let M (ε, Γ; W, P S ) be the maximum code size M for which trasmissio with average error probability of at most ε is possible through the chael W whe the state with distributio P S is kow at both ecoder ad decoder. The relatio of the radom variables is depicted i Figure.

2 M S Fig.. e X W Y S d M Casual structure whe state is kow at seder ad receiver. Whe we cosider uses of the chael, W P(Y X S ) observes the followig memoryless behavior W (y x, s) = W (y k x k, s k ), (x, y, s) X Y S. ad admissible iputs satisfy b s k (x k ) Γ. To model geeral behavior of the chael, we allow the state sequece or source Ŝ := S = (S (),..., S () ) = to evolve i a arbitrary maer i the sese of Verdú-Ha 2. We will also eed the followig defiitios of limits i probability. For a sequece of radom variables A =, the ad lim sup i probability 2, 3 are defied as p- p-lim sup A := sup r R : lim PrA < r = 0, A := if r R : lim PrA > r = 0. B. Defiitio ε-capacity ad its Dual ad We say that R R is a (ε, Γ)-achievable rate if there exists a sequece of o-egative umbers ε = such that log M (ε, Γ; W, PŜ) R, lim sup ε ε. The ε-capacity-cost fuctio C(ε, Γ; W, PŜ) for ε 0, ) is the supremum of all (ε, Γ)-achievable rates. The capacity-cost fuctio C(Γ; W, PŜ) := C(0, Γ; W, PŜ). Similarily, R R is a optimistic (ε, Γ)-achievable rate if there exists a sequece of o-egative umbers ε = for which log M (ε, Γ; W, PŜ) R, ε < ε. The optimistic ε-capacity-cost fuctio C (ε, Γ; W, PŜ) for ε (0, is the supremum of all optimistic (ε, Γ)-achievable rates. The optimistic capacity-cost fuctio C (Γ; W, PŜ) := C (, Γ; W, PŜ). Note that Γ C(ε, Γ; W, PŜ) ad Γ C (ε, Γ; W, PŜ) are cocave ad hece cotiuous for Γ > 0. A chael W with geeral state Ŝ has the strog coverse property 2, Sec. V 3, Def if C(Γ; W, PŜ) = C (Γ; W, PŜ), for all Γ > 0. This is the form of the strog coverse property stated i 7. I other words, the strog coverse property holds if ad oly if for every sequece ε = for which log M (ε, Γ; W, PŜ) > C(Γ; W, PŜ), we have lim ε =. We ote that eve though the quatities above are defied based o the average error probability, all the results i the followig also hold for the maximum error probability. III. MAIN RESULTS A. (ε, Γ)-Capacity ad its Dual I order to state the (ε, Γ)-capacity, it is coveiet to defie J(R Γ; W, PŜ) := lim sup Pr R (Γ). where the probability is take with respect to Ŝ. Note that PrR C S k (Γ) is the cdf of the radom variable C S k (Γ) (the iformatio spectrum 3) so J( Γ; W, PŜ) is the lim sup of this cdf. Theorem ((ε, Γ)-Capacity). For every ε 0, ), C(ε, Γ; W, PŜ) = sup R J(R Γ; W, PŜ) ε. () The case of most iterest is the capacity-cost fuctio. I this case, it is easy to check from the defiitio of J(R Γ; W, PŜ) ad the p- that () reduces to C(Γ; W, PŜ) = p- (Γ). (2) I the same way, i order to state our result for the optimistic (ε, Γ)-capacity, it is coveiet to defie the quatity: J (R Γ; W, PŜ) := R Pr (Γ). Note that J ( Γ; W, PŜ) is simply the of the cdf of the radom variable C S k. Theorem 2 (Dual (ε, Γ)-Capacity). For every ε (0,, C (ε, Γ; W, PŜ) = sup R J (R Γ; W, PŜ) < ε. (3) The case of most iterest correspods to the optimistic capacity-cost fuctio. I this case (3) reduces to C (Γ; W, PŜ) = p-lim sup (Γ). (4) Note that both the ε-capacity-cost fuctios are expressed solely i terms of the sequece of radom variables C S k (Γ). This is because the chael W is wellbehaved; it is memoryless ad statioary ad thus ca effectively be characterized by C s (Γ) for each state s S. However, the state process is geeral so, aturally, from the iformatio spectrum method 3, we eed to use probabilistic limit operatios. It turs out that these limits are applied to the Cèsaro mea of the capacity-cost fuctios C s (Γ) s S. B. Strog Coverse Uitig (2) ad (4) ad recallig the defiitio of the strog coverse property, we immediately obtai the followig: Theorem 3 (Strog Coverse). A ecessary ad sufficiet coditio for the strog coverse property to hold is p- (Γ) = p-lim sup (Γ). (5)

3 I other words, for the strog coverse to hold, the sequece of radom variables C S k (Γ) must coverge (poitwise i probability) to C(Γ) 0. Furthermore, C(Γ) is the capacity-cost fuctio as well as the optimistic capacity-cost fuctio of the chael W with geeral state Ŝ. While Theorem 3 provides a ecessary ad sufficiet coditio for the strog coverse to hold, it requires the full statistics of Ŝ. Thus (5) may be hard to verify i practice. We provide a simpler coditio for the strog coverse to hold that is based oly o first- ad secod-order statistics. Corollary 4 (Sufficiet Coditio for Strog Coverse). The strog coverse holds with capacity-cost fuctio C(Γ) 0 if the followig limits exist lim lim 2 E (Γ) = C(Γ) ad (6) l= Cov ( (Γ), C Sl (Γ)) = 0 (7) Observe that if the geeral source Ŝ decorrelates quickly such that Cov ( (Γ), C Sl (Γ)) is small for large lags k l, the the covariace coditio i (7) is likely to be satisfied. I the followig subsectio, we provide some examples for which the covariace coditio either holds or is violated. C. Examples I this sectio, we provide a few examples to illustrate the geerality of the model ad the strog coverse coditios. We assume that there are o cost costraits here. Example. Suppose the geeral source Ŝ := S = (S (),..., S () ) = is such that S m () = S for m, where S P S P(S). The, each covariace i (7) is equal to Var(C S ). If this variace is positive, the either the sufficiet coditio i Corollary 4 or the ecessary coditio i (5) is satisfied. For such a state sequece, which correspods to a mixed chael 3, Sec. 3.3, the ε-capacity is give by C(ε; W, PŜ) = supr PrR C S ε. Example 2. Suppose the source Ŝ is idepedet ad idetically distributed (i.i.d.) with commo distributio P S P(S). The, Cov (, C Sl ) = 0 for k l ad hece the sum i (7) is simply Var(C S ). This grows liearly i ad hece, the strog coverse coditio holds with C(W, PŜ) = max I(P, W P S). P P(X S) Example 3. Suppose that the source Ŝ evolves accordig to a time-homogeeous, irreducible Markov chai whose states are positive recurret. Such a Markov chai admits a uique statioary distributio π P(S). It is easy to check that Cov (, C Sl ) ae b k l for some a, b > 0. Also, EC S k s π(s)c s. Hece, both (6) ad (7) are satisfied ad the chael admits a strog coverse with C(W, PŜ) = max I(P, W π). P P(X S) A variatio of the Gilbert-Elliott chael 8 0 with state iformatio at the ecoder ad decoder is modeled i this way. I fact, sice the above capacity ca be achieved eve without state iformatio at the ecoder (by symmetry), the above implies the strog coverse for the regular Gilbert- Elliott chael. Example 4. The covariace coditio (7) is ot sufficiet i geeral. Cosider the memoryless (but o-statioary) source Ŝ give by S k J S k = S 2 k / J, where J := i N : 2 2k i < 2 2k, k N. Sice Ŝ is memoryless, just as i Example 2, Cov (, C Sl ) = 0 for k l. Hece, the covariace coditio is satisfied. However, C(W, PŜ) = p- C (W, PŜ) = p-lim sup = 2c 3 + d 3, ad = c 3 + 2d 3, where the parameters c := miec S, EC S2 ad d := maxec S, EC S2. If EC S EC S2, the c < d ad hece the ecessary ad sufficiet coditio i Theorem 3 is ot satisfied ad the strog coverse property does ot hold. A. Oe-Shot Bouds IV. PROOFS For the direct part, we require a state-depedet geeralizatio of Feistei s boud 3. The proof is omitted. Propositio 5 (State-Depedet Feistei Boud). Let Γ > 0 ad let P P(X S) be ay iput distributio. The, for ay η > 0, there exists a code C = M, e, d such that p max (C; W, P S ) Pri(X; Y S) log M + η + exp( η) + Prb S (X) > Γ. We prove a geeralizatio of our oe-shot coverse i 5 (see also refereces therei). Propositio 6 (State-Depedet Fuctio Coverse). Let 0 ε < ad let Γ > 0. The, for ay δ > 0, log M (ε, Γ; W, P S ) if Q P(Y S) sup x :S X sup R Pr jq (x(s); Y S) R ε + δ log δ. with x : S X satisfyig b s (x(s)) Γ for all s S. Proof: We cosider a geeral code M, e, d, where the ecoder is such that e(s, m) B s (Γ) for all m M, s S. Moreover, let Q P(Y S) be arbitrary. The followig iitial boud ca be derived followig the lies of 5. See also 4. Let m = log M ad δ > 0, the m sup R Pr jq (X; Y S) R ε + δ log δ.

4 Here, P XY S is iduced by the code applied to S P S ad a uiform M. I particular, ote that P X S (x s) = 0 if x / B s. We may expad the Prj Q (X; Y S) R as follows: P S (s) P X S (x s) Pr jq (x; Y s) R X =x, S =s. s S x B s Clearly, there exists a fuctio x Q : S X with x Q (s) arg mi x Bs(Γ) Prj Q (x; Y s) R X =x, S =s such that Prj Q (x Q(s); Y s) R S =s Prj Q (X; Y s) R S =s. Hece, we ca relax the coditio o the supremum to get m sup R Pr j Q (x Q(S); Y S) R ε+δ log δ, which cocludes the proof after first maximizig over fuctios x ad the miimizig over distributios Q. B. ε-capacity ad its Dual We igore cost costraits i the remaider to make the presetatio more cocise. We combie the proofs of Theorems ad 2, but cosider direct ad coverse bouds separately. I the proofs, we iitially fix N ad cosider the - fold chael W P(Y X S ) ad a state govered by P S P(S ). We use bold letters to deote strigs of legth, e.g., S for S, ad the shorthad Pr = Pr. We use the (coditioal) iformatio variace, defied i 5. I particular, uitig 5, Lem. 62 ad 3, Rmk. 3.., the followig uiform bouds hold V (P, W ) U(P, W ) 2.3 log Y. (8) For every x : S X, the joit type 6 of x(s) ad s S is defied as T x,s (x, s) := δ sk,sδ xk (s),x = T x s (x s)t s (s). (9) Note that x T x(x, s) = T s (s) is the type of s ad T x,s (x s) is well-defied if we set T x s (x s) to uiform for all s S such that T s (s) = 0. We collect all these coditioal types i a set V satisfyig V ( + ) X S. We also defie T x s W (y s) := x T x s(x s)w (y x, s) as the correspodig output distributio for every T x s V. ) Direct: I the followig, we show that for ε 0,, C(ε; W, PŜ) supr J(R W, PŜ) ε, ad (0) C (ε; W, PŜ) supr J (R W, PŜ) < ε. () Proof: Cosider the iput distributio P P(X S) that maximizes the coditioal mutual iformatio, i.e. P ( s) arg max P P(X ) I(P, W s ). We ow apply Propositio 5 to W with iput distributio P (x s) = P (x k s k ) ad fid that there exists a code C = M, e, d, with M = exp(r) for ay R > 0, which satisfies, for ay ν > 0, p max (C; W, P S ) Pr i(x; Y S) R + ν + exp( ν). =: p(r) The probability above ca be bouded as follows p(r) = P S (s) Pr i(x k ; Y k s k ) R + ν S=s s S ( P S (s) C sk R + 2ν s S + Pr i(x k ; Y k s k ) < ) C sk ν S=s 2.3 log Y Pr R + 2ν + ν 2, (2) where we employed Chebyshev s iequality ad (8) to get U(P ( s k ), W sk ) 2.3 log Y. To show (0), set R := supr J(R W, PŜ) ε 3ν ad ote that R is ε-achievable sice (2) implies lim sup p max (C; W, P S ) lim sup Pr R + 2ν ε, (3) where the last iequality is by defiitio of J(R W, PŜ). Sice this holds for all ν > 0, Eq. (0) follows. Eq. () follows aalogously by defiig R := supr J (R W, PŜ) < ε 3ν appropriately ad takig a i (3). 2) Coverse: I the followig, we show that for ε 0,, C(ε; W, PŜ) supr J(R W, PŜ) ε, ad (4) C (ε; W, PŜ) supr J (R W, PŜ) < ε. (5) Proof: For repetitios of the chael, we cosider the followig covex combiatio of coditioal distributios: Q () (y s) := V T x s V Usig this i Propositio 6 yields log M (ε; W, P S ) + log δ T x s W (y k s k ). sup sup R Pr j Q ()(x(s); Y S) R ε+δ, x:s X where we recall that the probability is evaluated for the distributio PrS=s, Y =y = P S (s)w (y x(s), s). Now, Pr j Q ()(x(s); Y S) R = P S (s) Pr log W (Y k x k (s), s k ) R S=s Q () (Y s) s S P S (s) Pr log W (Y k x k (s), s k ) S=s T s S x s W (Y k s k ) R log V = Pr j Tx S (x k (S); Y k S k ) R log V.

5 Thus, we fid the followig boud log M (ε; W, P S ) + log δ log V sup sup R Pr j Tx S (x k (S); Y k S k ) R ε+δ. x =: cv(x) We ext aalyze cv(x) for a fixed fuctio x : S X. We first employ Markov s iequality which states for ay positivevalued fuctio f: Ef(S) γ Prf(S) γ. We apply this for f(s) = Pr S=s... to the above ad fid cv(x) sup R P S Aγ (R) ε + δ, (6) γ where ε + δ < γ < ad A γ (R) S is the set A γ (R) := s Pr j Tx s (x k (s); Y k s k ) R S=s γ. Furthermore, cosider the set B γ (R) := s I(T V (T x s, W T s ) x s, W T s ) + R, γ where the expectatio ad variace are defied as E j Tx s (x k (s); Y k s k ) S = s = I ( T x s, W ) Ts, Var j Tx s (x k (s); Y k s k ) S = s = V ( T x s, W ) Ts, ad we employed the defiitio of the expected coditioal iformatio variace 5. By Chebyshev s iequality, B γ (R) A γ (R) for all R R. Furthermore, I ( T x s, W ) Ts = T s (s )I ( T x s ( s ) ), W s s S s S T s (s )C s = C sk. ad V is bouded usig (8). Substitutig this ito (6) yields cv(x) sup R Pr ε + δ 2.3 log Y +, γ γ which is idepedet of x : S X. Fially, the asymptotics (4) ad (5) ca be show as follows. Due to the above, ay ε-achievable rate R satisfies R sup M (ε ; W, P S ) R Pr ε for some sequece ˆε = ε = with lim sup ε ε ad ˆε defied via ε = (ε + δ)/γ with δ = / ad γ = / such that the limits of the sequeces coicide. Hece, for ay ξ > 0, there exists a costat N ξ N such that for all > N ξ, Pr C S k R ξ ε. Thus, i particular, lim sup Pr R ξ lim sup ε ε. (7) Hece, the supremum over all ε-achievable rates is upper bouded by the supremum of all R s satisfyig (7), cocludig the proof of the first statemet as ξ 0. The secod statemet (5) follows aalogously by choosig a sequece ˆε with ε < ε ad takig a i (7). C. Strog Coverse Proof: It remais to show that Corollary 4 holds. If the covariace coditio (7) is satisfied ad we apply Chebyshev s iequality appropriately, we obtai p-lim sup E p- lim sup E. Furthermore, if the limit i (6) exists, all iequalities above are equalities, which meas that the coditio of Theorem 3 is satisfied. Hece, the strog coverse property holds. REFERENCES G. Keshet, Y. Steiberg, ad N. Merhav, Chael codig i the presece of side iformatio. Foudatios ad Treds i Commuicatio ad Iformatio Theory, Now Publishers Ic, S. Verdú ad T. S. Ha, A geeral formula for chael capacity, IEEE Tras. o If. Th., vol. 40, pp , Apr T. S. Ha, Iformatio-Spectrum Methods i Iformatio Theory. Spriger Berli Heidelberg, Feb P.-N. Che ad F. Alajaji, Optimistic Shao codig theorems for arbitrary sigle-user systems, IEEE Tras. o If. Th., vol. 45, o. 7, pp , M. Tomamichel ad V. Y. F. Ta, A tight upper boud for the third-order asymptotics of discrete memoryless chaels, arxiv: cs.it, Nov M. Tomamichel ad V. Y. F. Ta, ε-capacities ad secod-order codig rates for chaels with geeral state, arxiv: cs.it, May M. Hayashi ad H. Nagaoka, Geeral Formulas for Capacity of Classical-Quatum Chaels, IEEE Tras. o If. Theory, vol. 49, pp , July E. N. Gilbert, Capacity of burst-oise chaels, Bell Syst. Tech. J., vol. 39, pp , Sep E. O. Elliott, Estimates of error rates for codes o burst-oise chaels, Bell Syst. Tech. J., vol. 42, pp , Sep Y. Polyaskiy, H. V. Poor, ad S. Verdú, Dispersio of the Gilbert- Elliott chael, IEEE Tras. o If. Th., vol. 57, o. 4, pp , 20. A. Feistei, A ew basic theorem of iformatio theory, IEEE Tras. o If. Th., vol. 4, o. 4, pp. 2 22, A. J. Thomasia, Error bouds for cotiuous chaels, i Fourth Lodo Symposium o Iformatio Theory,, pp , C. Cherry, Ed. Butterworth, J. N. Laema, O the Distributio of Mutual Iformatio, i Iformatio Theory ad Applicatios Workshop, L. Wag, R. Colbeck, ad R. Reer, Simple chael codig bouds, i Itl. Symp. If. Th., (Seoul, South Korea), Y. Polyaskiy, H. V. Poor, ad S. Verdú, Chael codig i the fiite blocklegth regime, IEEE Tras. o If. Th., vol. 56, pp , May I. Csiszár ad J. Körer, Iformatio Theory: Codig Theorems for Discrete Memoryless Systems. Cambridge Uiversity Press, 20.