On Evaluating the Rate-Distortion Function of Sources with Feed-Forward and the Capacity of Channels with Feedback.

Transcription

1 O Evaluatig the Rate-Distortio Fuctio of Sources with Feed-Forward ad the Capacity of Chaels with Feedback. Ramji Vekataramaa ad S. Sadeep Pradha Departmet of EECS, Uiversity of Michiga, A Arbor, MI 4805 rvekata@umich.edu, pradhav@eecs.umich.edu THIS PAPER IS ELIGIBLE FOR THE STUDENT PAPER AWARD Abstract We study the problem of computig the rate-distortio fuctio for sources with feed-forward ad the capacity for chaels with feedback. The formulas (ivolvig directed iformatio) for the optimal rate-distortio fuctio with feed-forward ad chael capacity with feedback are multiletter expressios ad caot be computed easily i geeral. I this work, we derive coditios uder which these ca be computed for a large class of sources/chaels with memory ad distortio/cost measures. Illustrative examples are also provided. I. INTRODUCTION Feedback is widely used i commuicatio systems to help combat the effect of oisy chaels. It is well-kow that feedback does ot icrease the capacity of a discrete memoryless chael []. However, feedback could icrease the capacity of a chael with memory. Recetly, directed iformatio has bee used to elegatly characterize the capacity of chaels with feedback [2], [3], [4]. The source codig couterpart to chael codig with feedback- source codig with feed-forward- has recetly bee studied i [5], [6], [7], [8]. The optimal rate-distortio fuctio with feed-forward was characterized usig directed iformatio i [6]. I this work, we study the problem of computig these ratedistortio ad capacity expressios. The formulas (ivolvig directed iformatio) for the optimal rate-distortio fuctio with feed-forward [6] ad chael capacity with feedback [4] are multi-letter expressios ad caot be computed easily i geeral. We derive coditios uder which these ca be computed for a large class of sources (chaels) with memory ad distortio (cost) measures. We also provide illustrative examples. Throughout, we cosider source feed-forward ad chael feedback with arbitrary delay. Whe the delay goes to, we obtai the case of o feed-forward/feedback. II. SOURCE CODING WITH FEED-FORWARD A. Problem Formulatio I simple terms, source codig with feed-forward is the source codig problem i which the decoder gets to observe some past source samples to help recostruct the preset This work was supported by NSF Grat ITR ad Grat (CA- REER) CCF sample. Cosider a geeral discrete source X with alphabet X ad recostructio alphabet X. The source is characterized by a sequece of distributios deoted by P X = {P X } =. There is a associated sequece of distortio measures d : X X R +. It is assumed that d (x, ˆx ) is ormalized with respect to ad is uiformly bouded i. For example d (x, ˆx ) may be the average per-letter distortio, i.e., d(x i, ˆx i ) for some d : X ˆX R +. Defiitio : A (N, 2 NR ) source code with delay k feedforward of block legth N ad rate R cosists of a ecoder mappig e ad a sequece of decoder mappigs g i, i =,..., N, e : X N {,..., 2 NR } g i : {,..., 2 NR } X i k X, i =,..., N. The ecoder maps each N-legth source sequece to a idex i {,..., 2 NR }. The decoder receives the idex trasmitted by the ecoder, ad to recostruct the ith sample, it has access to the source samples util time (i k). We wat to miimize R for a give distortio costrait. Defiitio 2: (Probability of error criterio) R is a ɛ- achievable rate at probability- distortio D if for all sufficietly large N, there exists a (N, 2 NR ) source codebook such that ( P X N x N : d N (x N, ˆx N ) > D ) < ɛ, ˆx N deotes the recostructio of x N. R is a achievable rate at probability- distortio D if it is ɛ-achievable for every ɛ > 0. We ow give a brief summary of the rate-distortio results with feed-forward foud i [6]. The rate-distortio fuctio with feed-forward (delay ) is characterized by directed iformatio, a quatity defied i [2]. The directed iformatio flowig from a radom sequece ˆX N to a radom sequece X N is defied as I( ˆX N X N ) = N I( ˆX ; X X ). () = Whe the feed-forward delay is k, the rate-distortio fuctio is characterized by the k delay versio of the directed

2 iformatio: I k ( ˆX N X N ) = N I( ˆX +k ; X X ). (2) = Whe we do ot make ay assumptio o the ature of the joit process {X, ˆX}, we eed to use the iformatio spectrum versio of (2). I particular, we will eed the quatity I k ( ˆX X) lim sup iprob P kˆx X = log P ˆXi ˆX i,x. i k P X, ˆX P kˆx X P X, (3) It should be oted that (2) ad (3) are the same whe the joit process {X, ˆX} is statioary ad ergodic. Theorem : [6] For a arbitrary source X characterized by a distributio P X, the rate-distortio fuctio with feedforward, the ifimum of all achievable rates at probability- distortio D, is give by R ff (D) = if I k( ˆX X), (4) P ˆX X :ρ(p ˆX X ) D ρ(p ˆX X ) lim sup d (x, ˆx ) iprob o (5) = if h : lim P X, ˆX ((x, ˆx ) : d (x, ˆx ) > h) = 0. B. Evaluatig the Rate-Distortio Fuctio with Feed-forward The rate-distortio formula i Theorem is a optimizatio of a multi-letter expressio: I k ( ˆX X) lim sup iprob log P X, ˆX P kˆx X P X, This is a optimizatio over a ifiite dimesioal space of coditioal distributios P ˆX X. Sice this is a potetially difficult optimizatio, we tur the problem o its head ad pose the followig questio: Give a source X with distributio P X ad a coditioal distributio P ˆX X, for what sequece of distortio measures does P ˆX X achieve the ifimum i the rate-distortio formula? A similar approach is used i [9] (Problem 2 ad 3, p. 47) to fid optimizig distributios for discrete memoryless chaels ad sources without feedback/feed-forward. It is also used i [0] to study the optimality of trasmittig ucoded source data over chaels ad i [] to study the duality betwee source ad chael codig. Give a source X, suppose we have a huch about the structure of the optimal coditioal distributio. The followig theorem (proof omitted) provides the distortio measures for which our huch is correct. The lim sup iprob of a radom sequece A is defied as the smallest umber α such that P (A α + ɛ) = 0 for all ɛ > 0 ad is deoted A. Theorem 2: Suppose we are give a statioary, ergodic source X characterized by P X = {P X } = with feedforward delay k. Let P ˆX X = {P X X } = be a coditioal distributio such that the joit distributio is statioary ad ergodic. The P ˆX X achieves the rate-distortio fuctio if for all sufficietly large, the distortio measure satisfies d (x, ˆx ) = c log P X, ˆX(x, ˆx ) P kˆx X (ˆx x ) + d 0(x ), (6) P kˆx X (ˆx x ) = P ˆXi X i k, ˆX (ˆx i i x i k, ˆx i ), c is ay positive umber ad d 0 (.) is a arbitrary fuctio. C. Markov Sources with Feed-forward A statioary, ergodic mth order Markov source X is characterized by a distributio P X = {P X } = P X = P Xi X i,. (7) i m Let the source have feed-forward with delay k. We first ask: Whe is the optimal joit distributio also mth order Markov i the followig sese: P X, ˆX = P Xi, ˆX i X i,. (8) i m I other words, whe does the optimizig coditioal distributio have the form P ˆX X = P ˆXi X. (9) i m, i The aswer, provided by Theorem 2, is stated below. I the sequel, we drop the subscripts o the probabilities to keep the otatio clea. Corollary : For a mth order Markov source (described i (7)) with feed-forward delay k, a mth order coditioal distributio (described i (9)) achieves the optimum i the rate-distortio fuctio for a sequece of distortio measures {d } give by d (x, ˆx ) = c X log P (x i, ˆx i x i i m ) P (ˆx i ˆx i i k+, xi k i k+ m ) + d0(x ), (0) c is ay positive umber ad d 0 (.) is a arbitrary fuctio. Proof: The proof ivolves substitutig (7) ad (9) i (6) ad performig a few maipulatios. I the followig sectio, we provide two examples to illustrate how Theorem 2 ca be used to determie the ratedistortio fuctio of sources with feed-forward.

3 -p 0 0 p 0 q 0 Fig.. A. Stock-market example -p -q p Markov chai represetig the stock value TABLE I DISTORTION e (ˆx i, x i = j, x i ) (x i, x i ) j, j + j, j j, j ˆx i = ˆx i = 0 III. EXAMPLES Suppose that we wish to observe the behavior of a particular stock i the stock market over a N day period. Assume that the value of the stock ca take k + differet values ad is modeled as a k + -state Markov chai, as show i Fig.. If o a particular day, the stock is i state i, i < k, the o the ext day, oe of the followig ca happe. The value icreases to state i + with probability p i. The value drops to state i with probability q i. The value remais the same with probability p i q i. Whe the stock-value is i state 0, the value caot decrease. Similarly, whe i state k, the value caot icrease. Suppose a ivestor ivests i this stock over a N day period ad desires to be forewared wheever the value drops. Assume that there is a isider (with a priori iformatio about the behavior of the stock) who ca sed iformatio to the ivestor at a fiite rate. The value of the stock is modeled as a Markov source X = {X }. The decisio ˆX of the ivestor is biary: ˆX = idicates that the price is goig to drop from day to, ˆX = 0 meas otherwise. Before day, the ivestor kows all the previous values of the stock X ad has to make the decisio ˆX. Thus feed-forward is automatically built ito the problem. The ivestor makes a error either whe he fails to predict a drop or whe he falsely predicts a drop. The distortio is modeled usig a Hammig distortio criterio as follows. d (x, ˆx ) = e(ˆx i, x i, x i ), () e(.,.,.) is the per-letter distortio give Table I. The miimum amout of iformatio (i bits/sample) the isider eeds to covey to the ivestor so that he ca predict drops i value with distortio D is deoted R ff (D). q k -q k k Propositio : For the stock-market problem described above, k R ff (D) = π i (h(p i, q i, p i q i ) h(ɛ, ɛ)) + π k (h(q k, q k ) h(ɛ, ɛ)), h() is the etropy fuctio, [π 0, π,, π k ] is the statioary distributio of the Markov chai ad ɛ = D π 0. Proof: We will use Corollary to verify that a first-order Markov coditioal distributio of the form P ˆX ˆX,X = P ˆX X,X, (2) achieves the optimum. Due to the structure of the distortio fuctio i Table I, we ca guess the structure of P (x i ˆx i, x i ) as follows. Whe X i = 0, the decoder ca always declare ˆX i = 0 - there is o error irrespective of the value of X i. So we assig P ( ˆX i = 0 x i = 0, x i = 0) = P ( ˆX i = 0 x i = 0, x i = ) =, which gives P (X i = 0 x i = 0, ˆx i = 0) = p. The evet (X i = 0, ˆX i = ) has zero probability. Thus we obtai the first two colums of Table II. Whe (X i = j, ˆX i = 0), 0 < j < k, a error occurs whe X i = j. This is assiged a probability ɛ. The remaiig probability ɛ is split betwee P (X i = j x i = j, ˆx i = 0) ad P (X i = j + x i = j, ˆx i = 0) accordig to their trasitio probabilities. I a similar fashio, we obtai all the colums i Table II. We ow show that the distortio criterio () ca be cast i the form d (x, ˆx ) = or equivaletly ( ) c log 2 P (x i ˆx i, x i )+d 0 (x i, x i ), (3) e(ˆx i, x i, x i ) = c log 2 P (x i ˆx i, x i ) + d 0 (x i, x i ), (4) thereby provig that the distributio i Table II is optimal. This is doe by substitutig values from Tables I ad II ito (4) to determie c ad d 0 (.,.). Sice the process {X, ˆX} is joitly statioary ad ergodic, the distortio costrait is equivalet to E[e(ˆx 2, x, x 2 )] D. To calculate the expected distortio E[e(ˆx 2, x, x 2 )] = P (x, x 2 )P (ˆx 2 x, x 2 ) e(ˆx 2, x, x 2 ), x,x 2,ˆx2 (5) we eed the (optimum achievig) coditioal distributio P ( ˆX 2 x, x 2 ). This is foud by substitutig the values from Tables I ad II i the relatio P (x 2 x, ˆx 2 ) = P (x 2 x )P (ˆx 2 x 2, x ) x 2 P (x 2 x )P (ˆx 2 x 2, x ). (6)

4 TABLE II THE DISTRIBUTION P (X i x i, ˆx i ) (x i, ˆx i ) 0, 0 0, j, 0 j, k, 0 k, x i = 0 p x i = p x i =.... x i = j ɛ ɛ ( ɛ)( p j q j ) ɛ( p j q j ) q j x i = j q j ( ɛ)p x i = j + j ɛp j q j q j x i =.... x i = k ɛ ɛ x i = k ɛ ɛ TABLE III THE CONDITIONAL DISTRIBUTION P ( ˆX i x i, x i ) (x i, x i ) 0, 0 0, j, j j, j j, j + k, k k, k ɛ( q ˆx i = 0 j ɛ) ( ɛ)( q j ɛ) ( ɛ)( q j ɛ) ɛ( q j ɛ) ( ɛ)( q j ɛ) q j ( 2ɛ) ( q j )( 2ɛ) ( q j )( 2ɛ) q j ( 2ɛ) ( q j )( 2ɛ) ( ɛ)(q ˆx i = 0 0 j ɛ) ɛ(q j ɛ) ɛ(q j ɛ) ( ɛ)(q j ɛ) ɛ(q j ɛ) q j ( 2ɛ) ( q j )( 2ɛ) ( q j )( 2ɛ) q j ( 2ɛ) ( q j )( 2ɛ) Thus we obtai the coditioal distributio P ( ˆX 2 x, x 2 ) show i Table III. Usig this i (5), we get E[e(ˆx 2, x, x 2 )] = ( π 0 )ɛ D (7) We ca ow calculate the rate distortio fuctio as R ff (D) = P (x 2 x, ˆx 2 ) P (x, x 2, ˆx 2 ) log 2 P (x 2 x ) x,x 2,ˆx 2 = H(X 2 X ) H(X 2 ˆX 2, X ) to obtai the expressio i Propositio. B. Gauss-Markov Source (8) Cosider a statioary, ergodic, first-order Gauss-Markov source X with mea 0, correlatio ρ ad variace σ 2 : X = ρx + N,, (9) {N } are idepedet, idetically distributed Gaussia radom variables with mea 0 ad variace ( ρ 2 )σ 2. Assume the source has feed-forward with delay ad we wat to recostruct at every time istat the liear combiatio ax +bx, for ay costats a, b. We use the mea-squared error distortio criterio: d (x, ˆx ) = (ˆx i (ax i + bx i )) 2. (20) The feed-forward distortio-rate fuctio for this source with average mea-squared error distortio was give i [5]. The feed-forward rate-distortio fuctio ca also be obtaied usig Theorem 2 (proof omitted): R ff (D) = 2 log σ2 ( ρ 2 ) D/a 2. (2) We must metio here that the rate-distortio fuctio i the first example caot be computed usig the techiques i [5]. IV. CHANNEL CODING WITH FEEDBACK I this sectio, we cosider chaels with feedback ad the problem of evaluatig their capacity. A chael is defied as a sequece of probability distributios: PY X ch ch = {PY X,Y } =. (22) I the above, X ad Y are the chael iput ad output symbols at time, respectively. The chael is assumed to have k delay feedback ( k < ). This meas at time istat, the ecoder has perfect kowledge of the chael outputs util time k to produce the iput x. The iput distributio to the chael is deoted by P k X Y = {P X X,Y k} =. I the sequel, we will eed the followig product quatities correspodig to the chael ad the iput. P ch Y X P k X Y P Yi X i,y i, (23) P Xi X i,y i k. The joit distributio of the system is give by P X,Y = {P X,Y } =, P X,Y = P X k ch Y P Y X. (24) Defiitio 3: A (N, 2 NR ) chael code with delay k feedforward of block legth N ad rate R cosists of a sequece of ecoder mappigs e i, ı =,..., N ad a decoder g, e i : {,..., 2 NR } Y i k X, i =,..., N g : Y N {,..., 2 NR }

5 Thus it is desired to trasmit oe of 2 NR messages over the chael i N uits of time. There is a associated cost fuctio for usig the chael give by c N (X N, Y N ). For example, this could be the average power of the iput symbols. If W is the message that was trasmitted, the the probability of error is P e = P r(g(y N ) W ). Defiitio 4: R is a (ɛ, δ)-achievable rate at probability- cost C if for all sufficietly large N, there exists a (N, 2 NR ) chael code such that P e < ɛ, P r(c N (X N, Y N ) > C) < δ R is a achievable rate at probability- cost C if it is (ɛ, δ)- achievable for every ɛ, δ > 0. Theorem 3: [6], [4] For a arbitrary chael PY X ch, the capacity with k delay feedback, the ifimum of all achievable rates at probability- cost C, is give by 2 ad C fb (C) = sup I(X Y ), (25) P k X Y :ρ(p k X Y ) C I(X Y ) lim if iprob P ch log Y X P Y ρ(px Y) k lim sup c (X, Y ) iprob = if{h : lim PX Y ((x, y ) : c (x, y ) > h)} = 0. I the above, we ote that P Y = P X,Y = P k ch X Y P Y X. X X A. Evaluatig the Chael Capacity with Feedback The capacity formula i Theorem 3 is a multi-letter expressio ivolvig optimizig the fuctio I(X Y ) over a ifiite dimesioal space of iput distributios PX Y k. Just like we did with sources, we ca pose the followig questio: Give a chael PY X ch ad a iput distributio PX Y k, for what sequece of cost measures does P X Y k achieve the supremum i the capacity formula? The followig theorem (proof omitted) provides a aswer. Theorem 4: Suppose we are give a chael PY X ch with k delay feedback ad a iput distributio PX Y k such that the joit process P X,Y give by (24) is statioary, ergodic. The the iput distributio PX Y k achieves the k delay feedback capacity of the chael if for all sufficietly large, the cost measure satisfies c (x, y ) = λ P ch log Y X (y x ) P Y (y + d 0, (26) ) λ is ay positive umber ad d 0 is a arbitrary costat. 2 The lim if iprob of a radom sequece A is defied as the largest umber α such that P (A α ɛ) = 0 for all ɛ > 0 ad is deoted A. B. Markov chaels with feedback The problem of evaluatig the capacity of fiite state machie chaels was studied recetly i [2] ad [3]. I [2], it was show that the capacity of such a chael is achieved by a feedback depedet Markov source, i.e., the optimal iput distributio is of the form {P X X,Y }, i.e., Markov i X but depeds o all the past Y symbols. We cosider a simple Markov chael with feedback delay ad the problem of evaluatig its capacity. The chael we study is characterized by PY ch X,Y = P ch Y X,Y. (27) Let the chael have feedback with delay. We are iterested i fidig cost measures for which the capacity of the chael i (27) is easily evaluated. We first ask: Whe is the optimal joit distributio first order Markov i the followig sese: P X,Y = P Xi,Y i Y i,. (28) I other words, whe does the optimizig iput distributio to have the form P X Y = P Xi Y i,. (29) From Theorem 4, it is see that this happes whe the costfuctio has the form: c (x, y ) = λ P Y ch log i X i,y i (y i x i, y i ) + d 0. P Yi Y i (y i y i ) (30) REFERENCES [] C. E. Shao, The zero-error capacity of a oisy chael, IRE Tras. If. Theory, vol. IT-2, pp. 8 9, 956. [2] J. Massey, Causality, Feedback ad Directed Iformatio, Proc. 990 Symp. o If. Theory ad Applicatios (ISITA-90), pp , 990. [3] G. Kramer, Directed Iformatio for chaels with Feedback. PhD thesis, Swiss Federal Istitute of Techology, Zurich, 998. [4] S. Tatikoda, Cotrol Uder Commuicatios Costraits. PhD thesis, Massachusetts Ist. of Techology, Cambridge, MA, September [5] T. Weissma ad N. Merhav, O competitive predictio ad its relatio to rate-distortio theory, IEEE Tras. If. Theory, vol. IT-49, pp , December [6] R. Vekataramaa ad S. S. Pradha, Directed Iformatio for Commuicatio problems with Side-iformatio ad Feedback/Feedforward, Proc. 43rd Aual Allerto Cof. (Moticello, IL), [7] E. Martiia ad G. W. Worell, Source Codig with Fixed Lag Side Iformatio, Proc. 42d Aual Allerto Cof. (Moticello, IL), [8] S. S. Pradha, Source codig with feedforward: Gaussia sources, i Proc. IEEE It. Symp. o If. Theory, p. 22, Jue [9] I. Csisz ar ad J. Korer, Iformatio Theory: Codig Theorems for Discrete Memoryless Systems. New York: Academic Press, 98. [0] M. Gastpar, B. Rimoldi, ad M. Vetterli, To code, or ot to code: lossy source-chael commuicatio revisited., IEEE Tras. If. Theory, vol. 49, o. 5, pp , [] S. S. Pradha, J. Chou, ad K. Ramchadra, Duality betwee source codig ad chael codig ad its extesio to the side-iformatio case, IEEE Tras. If. Theory, vol. 49, pp , May [2] S. Yag, A. Kavcic, ad S. Tatikoda, Feedback capacity of fiite-state machie chaels., IEEE Tras. If. Theory, vol. 5, o. 3, pp , [3] J. Che ad T. Berger, The capacity of fiite-state markov chaels with feedback., IEEE Tras. If. Theory, vol. 5, o. 3, pp , 2005.