ECE 729 Introduction to Channel Coding

Transcription

1 chaelcodg.tex May 4, 2006 ECE 729 Itroducto to Chael Codg Cotets Fudametal Cocepts ad Techques. Chaels Ecoders Code Rates Decoders Probablstc Model The Probablty of Error ew Codes from Old Codes by Throwg Away Codewords The Radom Codg Argumet The Codebook Reducto Argumet A Uderlyg Observato Puttg It All Together Costructo of Decoders Bouds o the Probablty of Error Codeword Costrats Achevable Rates 5 2. Determstc Codes Radom Codes Coectg Radom Codes ad Determstc Codes Capacty Regos Capacty-Cost Fuctos Cost per Bt Waveform Chaels ad Spectral Effcecy.. 7. Fudametal Cocepts ad Techques.. Chaels Gve a set X of chael put symbols, a set Y of chael output symbols, ad a postve teger, a chael W s a trasto probablty from X to Y. I other words, f X (X,...,X ) s a X -valued radom varable ad Y (Y,...,Y ) s a Y -valued radom varable, the W (B x) P(Y B X x), B Y, s the codtoal probablty that the radom sequece Y of chael output symbols les B gve that the chael put sequece s X x. x W If Y s a fte set, the so s Y, ad W s gve by a codtoal probablty mass fucto, whch we also deote by W, ad we have W (B x) W (y x). y B Y I most applcatos, W s ot explctly gve, but s defed mplctly. Example. Let X Y deote the tegers {0,...,m } uder mod-m addto. Let the compoets of Z : (Z,...,Z ) be..d. X-valued radom varables wth commo pmf p Z (z) P(Z z) for z X. Let X be depedet of Z, ad put Y : X+Z, where the compoetwse addto s mod-m. The W (y x) : P(Y y X x) P(X+Z y X x) P(Z y x) p Z (y k x k ). If Y IR, the we take W to be gve by a codtoal probablty desty fucto, whch we deote by w, ad we have W (B x) w (y x)dy. B Here too, w s usually ot gve explctly. Example 2. Let X Y IR. Let the compoets of Z : (Z,...,Z ) be..d. real-valued radom varables wth commo probablty desty p Z (z). Let X be depedet of Z, ad put Y : X+Z. The P(Y y X x) P(X+Z y X x) P(Z y x) P(Z k y k x k ) yk x k Dfferetatg wth respect to the y k, we have.2. Ecoders w (y x) p Z (y k x k ). p Z (z k )dz k. A mappg f :{,...,} X s called a ecoder. The tegers,..., are called messages. The correspodg codewords are deoted by x : f(). Thus, each x X. The collecto (x,...,x ) s the correspodg codebook. Kowg the codebook (x,...,x ) s the same as kowg the ecoder f, ad we sometmes just wrte f (x,...,x ), whch s a elemet of (X ).

2 chaelcodg.tex May 4, 2006 message f ecoder x codeword However, ths problem s ot well defed uless we specfy the jot dstrbuto of M ad Y. We take.2.. Code Rates If there are 2 k messages, the each message ca be thought of as a k-bt word. Eve for ot a power of 2, we thk of each message as cosstg of log 2 bts. If we use atural logarthms, we say that each message cossts of l ats. We ofte do ot specfy the base of the logarthm ad just wrte log. I ths case, we use exp for the verse of log. Hece, f t s uderstood that the logarthm base s b, the exp(x) b x. I partcular, f the logarthm base s 2, the exp(x) 2 x, ad f the logarthm base s e, the exp(x) e x. Gve a code f :{,...,} X, or equvaletly a codebook (x,...,x ), the rate of the code s R log. () I ths expresso, s the umber of chael uses or chael symbols trasmtted for each message. If the logarthm base s 2, the the umerator has uts of bts, ad the quotet R has uts of bts per chael use or bts per chael symbol. If the chael trasmts R c chael symbols per secod, the R b : R c log 2 has uts of bts per secod. Combg ths wth () yelds.3. Decoders (2) R b R c R. (3) A mappg ϕ:y {,...,} s called a decoder. If for,..., we put D : {y : ϕ(y) }, the the D form a partto of Y sce they are dsjot ad ther uo s Y. We call the D decodg sets..4. Probablstc Model y ϕ decoder If we apply the output of the ecoder to the chael, ad we apply the output of the chael to the decoder, the we obta the chael codg system show Fg.. otce that the f ecoder x W chael Y î ϕ decoder Fgure. A chael codg system. put to the system s ad the output s î. We hope that most of the tme î. To make ths more precse, we assume that the put to the ecoder s a {,...,}-valued radom varable M, ad we are terested the probablty of error P(ϕ(Y) M). î P(M,Y B) : W (B x ),,...,, B Y. (4) otce that sce W (Y x ), (4) mples that P(M ) /;.e., the radom message to be set s chose uformly from the message set {,...,}. It the follows that P(Y B M ) P(M,Y B) P(M ) W (B x ). (5) We also pot out that the defto of P (4) depeds o the codebook (x,...,x ). Hece, P(ϕ(Y) M) depeds o the codebook ad o the decoder ϕ..5. The Probablty of Error It s ow coveet to derve a somewhat explct formula for P(ϕ(Y) M). We use the law of total probablty ad substtuto to wrte P(ϕ(Y) M) P(ϕ(Y) M M )P(M ) P(ϕ(Y) M ) P(Y {y : ϕ(y) } M ) W ({y : ϕ(y) } x ), by (5). (6) Although the reader may have reservatos about our choosg to have M be uformly dstrbuted, may cases, we wll be able to obta a boud o the terms (6) that does ot deped o, say W ({y : ϕ(y) } x ) < λ,,...,. (7) If we ca establsh such a boud, the for ay probablty mass fucto q() o {,...,}, f we replace (4) wth P(M,Y B) : q()w (B x ),,...,, B Y, t wll follow that P(ϕ(Y) M) < P(ϕ(Y) M M )P(M ) P(ϕ(Y) M )q() W ({y : ϕ(y) } x )q() λq() λ. 2

3 chaelcodg.tex May 4, ew Codes from Old Codes by Throwg Away Codewords Let f :{,...,} X be ay ecoder, ad let ϕ:y {,...,} be ay decoder. The D : {y : ϕ(y) } cotas exactly those outputs y that are decoded to message. If G s ay subset of {,...,}, we defe the modfed ecoder f G :G X by f G () : f() x, G, ad we defe the modfed decoder ϕ G :Y G by ϕ(y), y D ϕ G (y) : G 0, otherwse, where 0 ca be ay fxed elemet of G. Hece, for G wth 0, ϕ G (y) y D, ad we ca wrte {y : ϕ G (y) } {y : ϕ(y) }, for G, 0. (8) Furthermore, f we put ( ) c H : D, G the ϕ G (y) 0 y D 0 H. We ca therefore wrte {y : ϕ G (y) } D c {y : ϕ(y) }, for all G. (9) Suppose that stead of (7), we have oly W ({y : ϕ(y) } x ) < λ, G, where G s a proper subset of {,...,}. Let f G ad ϕ G be the ecoder ad decoder just descrbed. The by (9), we ca wrte W ({y : ϕ G (y) } x ) W ({y : ϕ(y) } x ) < λ, G..7. The Radom Codg Argumet Let g be a oegatve fucto defed o some set Z, ad let Z be a Z-valued radom varable such that Eg(Z)] < λ. The there s at least oe z Z wth g(z) < λ. To see ths, suppose otherwse that g(z) λ for all z. The we would have Eg(Z)] λ, whch cotradcts the hypothess that Eg(Z)] < λ. We employ the radom codg argumet as follows. Lookg back at (6), we put e (x,...,x ) : W ({y : ϕ(y) } x ). If we ca fd a radom codebook (X,...,X ) such that Ee (X,...,X )] < λ, the there must be at least oe codebook (x,...,x ) (X ) wth e (x,...,x ) < λ..8. The Codebook Reducto Argumet.8.. A Uderlyg Observato Suppose θ,...,θ are oegatve umbers such that θ < λ. Put G : { : θ < 2λ}. We clam that G > /2. To see ths, wrte λ > θ ( ) θ + θ G c G c θ G c 2λ 2λ Gc. It follows that G c < /2, ad the G G G c > /2 / Puttg It All Together Suppose we ca fd a radom codebook (X,...,X ) such that Ee (X,...,X )] < λ. The by the radom codg argumet, there s a codebook (x,...,x ) such that e (x,...,x ) W ({y : ϕ(y) } x ) < λ. By the observato above, there s a subset G of {,...,} such that W ({y : ϕ(y) } x ) < 2λ, G. By the throwg away the codewords x for / G, ad usg the costructo Secto.6, we have a ecoder f G ad decoder ϕ G such that W ({y : ϕ G (y) } x ) < 2λ, G. (0) ote that the rate of ths modfed code satsfes log G > log/2 log log2. Hece, eve though we dscard half of the orgal codewords, the rate of the modfed code s early (log)/ for large..9. Costructo of Decoders Let B,...,B be subsets of Y, ad suppose we wat to have a decoder that aouces message whe t observes a chael output sequece y B. There are two dffcultes wth ths descrpto. Frst, f the B are ot dsjot ad y B B j, does the decoder aouce message or message j? Secod, f the decoder observes a y that does ot belog to ay of the B, what should the decoder do? 3

4 chaelcodg.tex May 4, 2006 Here s a stadard approach to address these dffcultes. Frst, f y belogs to more tha oe B, aouce the smallest correspodg value of. Secod, f y does ot belog to ay B, aouce some fxed message, say. To state ths approach mathematcally, let There are two specal cases to be metoed. Frst, f the B are dsjot, the F B, ad we have from (2) that {y : ϕ(y) } B c,,...,. Secod, f the uo of the B s equal to Y, the F : B F : B B c Bc, 2,...,, B c B j. ad put ( ) c F : F F c Fc. The F F for,...,, ad F F F Y. I other words, F,...,F ad F costtute a partto of Y. If we put ϕ(y) : I F (y)+i F (y), the for,...,, ϕ(y) y F, ad thus ϕ(y) y F c,,...,. () Furthermore, sce ϕ(y) y F F, we have ϕ(y) y F c F c, where t s mportat to ote that Hece, F c F c F c. ϕ(y) y F c,,...,. (2) To coclude, observe that B c, F c (, B c B j ), 2,...,. j< F c (3) We ca ow wrte ( B c B j ),,...,. (4) From (2) ad (4), we have ( {y : ϕ(y) } B c B j ),,...,. (5) ad from () ad (3) we have {y : ϕ(y) } F c B c,,...,. (6) It the follows from (5) that {y : ϕ(y) } B j,,...,. (7).0. Bouds o the Probablty of Error We ca use (5) to get a boud o a typcal term (6). We have W ({y : ϕ(y) } x ) W (B c x )+ W (B j x ). (8) By appealg to the two specal cases above, we see that f the B are dsjot, the we ca omt the sum o the rght-had sde. If the uo of the B s equal to Y, the we ca omt the term W (B c x ). Example 3 (Maxmum-Lkelhood Decodg). I maxmumlkelhood decodg, we take B {y : W (y x ) W (y x j ) for all j}. Sce every y must belog to at least oe of these sets, ther uo s Y. Lettg we have D : {y : ϕ(y) }, W ({y : ϕ(y) } x ) I D c (y)w (y x ). (9) y ow, D depeds o x,...,x. Let X,...,X be..d. wth commo pmf p (x) o X. The for ay 0 < ρ, ( ) EI D c (y) X x] P B j X x, by (7), ρ P(B j X x)], by, p. 36]. We ext observe that for ay s > 0, ( ) P(B j X x) P W (y X j ) W (y x) X x ( ) P W (y X j ) W (y x) p (x )I {W (y x ) W (y x)} x p (x W (y x ] ) s ). x W (y x) 4

5 chaelcodg.tex May 4, 2006 We ca ow wrte EI D c (y) X x] x p (x ) W (y x ] ) s ] ρ W (y x) ( ) ρ W (y x) ρs ] ρ p (x )W (y x ) s. x It ow follows that f x,...,x (9) are replaced by X,...,X ad we take the expectato, t s upper bouded by ] ρ ( ) ρ p (x)w (y x) ] ρs p (x )W (y x ) s. y x x Specalzg to s /(+ρ), we obta ] +ρ ( ) ρ p (x)w (y x) /(+ρ). (20) y x ote that sce the expectato of (9) s at most oe, (20) holds eve for ρ 0. I the case of a dscrete memoryless chael, W W. If we also take p p, the the above sum over x becomes ( p(x)w(y k x) ). /(+ρ) x Deotg ths product by λ(y ) λ(y ), (20) becomes ( ) ρ λ(y ) +ρ λ(y ) +ρ. y Hece, for a DMC wth p p, we have ] } +ρ ( ) { ρ p(x)w(y x) /(+ρ). y x If e R e R, we obta the further upper boud where e E o(ρ,p) ρr], ] +ρ E o (ρ, p) : l p(x)w(y x) /(+ρ). y x To obta the best boud, we optmze over ρ ad p. Hece, the best boud s e E r(r), where E r (R) : sup supe o (ρ, p) ρr] 0 ρ p s called the radom codg expoet... Codeword Costrats If a s oegatve cost fucto defed o X, we may requre that all codewords satsfy a (x) : a(x k ) A If A max : sup x X a(x) s fte, ad f A A max, the all codewords satsfy a (x) A, ad the costrat s sad to be actve. If A < A m : f x X a(x), the o codewords satsfy a (x) A. for some costat A. I other words, the total cost to trasmt a codeword caot exceed A. Whe X IR, we usually take a(x) x 2 as a measure of the eergy requred to sed the symbol x, ad a (x) s the average power used to trasmt the codeword. If we defe a decoder ϕ as outled Secto.9, we wll choose the set B to have the form B {y : expresso volvg x ad y ad a (x ) A}. Ths structure mples that B c {y : a (x ) > A}. (2) The set o the rght-had sde s ether Y the empty set accordace wth whether the codto a (x ) > A s true or false. Hece, f we ca show that > W ({y : a (x ) > A} x ), the the set s empty;.e., we must have a (x ) A. ow suppose that we proceed as Secto.8.2 wth some λ < /2. The > W ({y : ϕ G (y) } x ), G, by (0), W ({y : ϕ(y) } x ), by (8) f 0, W (B c x ), by (6) f, W ({y : a (x ) > A} x ), by (2). Hece, there are at least G 2 codewords that satsfy both (0) ad a (x ) A. Throwg away oe or two codewords f ecessary, we obta a modfed ecoder f G ad a modfed decoder ϕ G that satsfy W ({y : ϕ G (y) } x ) < 2λ for G ad a (x ) A for all G. Furthermore, the rate of ths code satsfes, for large, log G log( G 2 ) log G /2 log/4 log log Determstc Codes 2. Achevable Rates Defto D-A. A umber R 0 s achevable usg determstc codes uder the average probablty-of-error crtero ad codeword costrat A f for every λ > 0, for every R > 0, for all suffcetly large, there s a postve teger wth log > R R, ad there s a codebook (x,...,x ) ad decoder ϕ (usually depedg o the codebook) such that a (x ) A for all,...,, ad W ({y : ϕ(y) } x ) < λ. 5

6 chaelcodg.tex May 4, 2006 Defto D-M. A umber R 0 s achevable usg determstc codes uder the maxmal probablty-of-error crtero ad codeword costrat A f for every λ > 0, for every R > 0, for all suffcetly large, there s a postve teger wth log > R R, ad there s a codebook (x,...,x ) ad decoder ϕ (usually depedg o the codebook) such that a (x ) A for all,...,, ad W ({y : ϕ(y) } x ) < λ,,...,. It s easy to see that f a rate satsfes Defto D-M, the t satsfes Defto D-A. I other words, f C D-M (A) deotes the set of achevable rates uder Defto D-M, ad f C D-A (A) deotes the set of achevable rates uder Defto D-A, the 2.2. Radom Codes C D-M (A) C D-A (A). (22) Defto R-A. A umber R 0 s achevable usg radom codes uder the average probablty-of-error crtero ad codeword costrat A f for every λ > 0, for every R > 0, for all suffcetly large, there s a postve teger wth log > R R, ad there s a radom codebook (X,...,X ) ad decoder ϕ (usually depedg o the codebook) such that Ea (X )] A for all,...,, ad E ] W ({y : ϕ(y) } X ) < λ. Defto R-M. A umber R 0 s achevable usg radom codes uder the maxmal probablty-of-error crtero ad codeword costrat A f for every λ > 0, for every R > 0, for all suffcetly large, there s a postve teger wth 2.3. Coectg Radom Codes ad Determstc Codes By the radom codg argumets Sectos.8.2 ad., C R-A (A) C D-M (A). (24) We ext pot out that sce a determstc codebook s a specal case of a radom codebook, ad C D-M (A) C R-M (A) (25) C D-A (A) C R-A (A). (26) If we combe (23), (24), ad (25), we fd that C R-M (A) C R-A (A) C D-M (A). Furthermore, f we combe (24), (22), ad (26), we fd that Hece, C R-A (A) C D-M (A) C D-A (A). C R-M (A) C R-A (A) C D-M (A) C D-A (A). (27) 2.4. Capacty Regos The capacty rego uder a gve probablty-of-error crtero ad uder a costrat (f ay) s the set of all achevable rates uder the correspodg defto. As we have show, the four capacty regos above are all the same. We just do ot kow what ay of them s yet! A mportat property of capacty regos s that they are closed. Suppose R k s a sequece of achevable rates ad that R k coverges to some R. We must show that the lmt R s also achevable. Let R > 0 be gve. Sce R k R, there s some k 0 such that for all k k 0, R k > R R/2. ow, sce R k0 s achevable, there s a code wth log > R R, log > R k0 R/2, ad there s a radom codebook (X,...,X ) ad decoder ϕ (usually depedg o the codebook) such that Ea (X )] A for all,...,, ad E W ({y : ϕ(y) } X ) ] < λ,,...,. It s easy to see that f a rate satsfes Defto R-M, the t satsfes Defto R-A. I other words, f C R-M (A) deotes the set of achevable rates uder Defto R-M, ad f C R-A (A) deotes the set of achevable rates uder Defto R-A, the C R-M (A) C R-A (A). (23) satsfyg the codeword costrat (f ay), ad havg small probablty of error. Sce R k0 > R R/2, we also have log > R R. Aother mportat property of capacty regos s that f R s a achevable rate, the ay R < R s also achevable. Ths follows from the fact that R > R mples log > R R > R R. 6

7 chaelcodg.tex May 4, A A/ log 0 (A/) Fgure 2. A typcal capacty-cost fucto Capacty-Cost Fuctos Let C (A) deote ay of the four capacty regos (27), ad let deote the supremum of C (A). The s called the capacty uder put costrat A or t s called the capactycost fucto. From the foregog observatos, the capacty rego s the terval C (A) 0,]. As oted earler (Secto.), for A < A m, o codeword ca satsfy a (x) A. Hece, for such A there are o achevable rates, ad the capacty rego s empty. Although we may put C (A) : whe the capacty rego s empty, we usually just restrct atteto to A A m. However, f X s a fte set, t may happe that there s o x X that acheves the fmum A m : f x X a(x). I ths case, o codeword ca satsfy a (x) A m ad the capacty rego s empty. I these cases, t s uderstood that we would restrct atteto to A > A m. Capacty-cost curves are always odecreasg as show Fg. 2. Ths ca be establshed mathematcally as follows. From the defto of achevable rate, t s clear that A A 2 mples C (A ) C (A 2 ), whch the mples C(A ) C(A 2 ). Hece, s odecreasg. We also pot out that f A max : sup x X a(x) s fte, the becomes costat for large A; partcular, for A A max, C(A max ). Ths follows because, as oted Secto., f A A max, the all codewords satsfy a (x) A Cost per Bt If all codewords satsfy the costrat a(x k ) A, (28) the the cost to sed a message s at most A. Sce each message cossts of log 2 bts, we call A b : A log 2 A, by (), (29) R the average cost per bt. If s the hghest rate R at whch we ca relably trasmt messages usg codewords that satsfy the costrat, the the average cost per bt s lower bouded by the mmum average cost per bt 4] A b,m (A) : A. (30) Fgure 3. Graph of parametrc curve (A b,m (A),) (left) ad the parametrc curve (Ab,m db (A),) (rght) based o the capacty-cost fucto Fg. 2. The parametrc curve (A b,m (A),) at the left Fg. 3 shows the tradeoff betwee the mmum average cost per bt ad the correspodg maxmum relable trasmsso rate usg codewords that satsfy the costrat. To express the cost per bt db, put A db b,m (A) : 0log 0 A b,m(a) 0(log 0 e)la l]. We ca the plot the ew parametrc curve (Ab,m db (A),) as show at the rght Fg. 3. The slope of ths curve at a pot (A),) s (A db b,m d da d da A b,m db (A) C (A) ]. 0(log 0 e) A C (A) To compute the lmtg slope as A A m whe A m 0, we carefully wrte (cf. 2, p. 257]) lm A 0 A C (A) Hece, A lm A 0 C (A)A +AC (A) lm A 0 C, by l Hôptal s rule, (A)A 2 C (0) C (0). d lm A 0 d da A b,m db da 2C (0)2 (A) 0(log 0 e)c (0). (3) 2.6. Waveform Chaels ad Spectral Effcecy Gve a set of messages {,..., }, suppose we assocate correspodg sgals ξ (t),...,ξ (t) belogg to some fte-dmesoal space spaed by orthoormal waveforms ψ (t),...,ψ (t). The every sgal ca be expressed the form where ξ (t) x k : x k ψ k (t), ξ (t)ψ k (t)dt, 7

8 chaelcodg.tex May 4, 2006 ad the overbar deotes the complex cojugate f complexvalued waveforms are allowed. Whe trasmttg sgal ξ (t), the recever sees the waveform Z(t); e.g., Z(t) ξ (t) +V(t), where V(t) s addtve whte Gaussa ose (AWG). I ay case, we arbtrarly put Y : (Y,...,Y ), where 2 Y k : Z(t)ψ k (t)dt, k,...,. Whe the waveforms ψ all have durato T, to trasmt each message requres sedg a sgal ξ, whch s equvalet to sedg the chael symbols x,,...,x,. Hece, we the umber of chael uses per secod (cf. eq. (2)) s R c T. It s coveet to defe the badwdth of the sgals as 3, pp ] B : /(2T). Ths mples R c 2B. Wth ths substtuto (2), t follows that mmum average cost per bt db, deoted by α, we have the lear approxmato where C 2C (0) 2 0(log 0 e)c (0) (α α 0), α 0 : lm Ab,m db (A) 0log 0 A 0 Refereces C (0). ] R. G. Gallager, Iformato Theory ad Relable Commucato. ew York: Wley, ] K. Lu, V. Raghava, ad A. M. Sayeed, Capacty scalg ad spectral effcecy wde-bad correlated MIMO chaels, IEEE Tras. Iform. Theory, vol. 49, o. 0, pp , Oct ] R. J. McElece, The Theory of Iformato ad Codg. Readg, MA: Addso-Wesley, ] S. Verdú, O chael capacty per ut cost, IEEE Tras. Iform. Theory, vol. 36, o. 5, pp , Sept ] S. Verdú, Spectral effcecy the wdebad regme, IEEE Tras. Iform. Theory, vol. 48, o. 6, pp , Jue R b 2B log. The fracto o the left s called the spectral effcecy. It has uts of bts per secod per Hertz. From the rght-had sde, we see that f there s a put costrat as Secto 2.5, the the maxmum spectral effcecy of a relable system s. I waveform chaels, there s usually a costrat o the average power (eergy per secod). For sgals of durato T, ths meas T ξ (t) 2 dt P. T 0 By Parseval s formula, T 0 ξ (t) 2 dt x k 2. Hece, f we take a(x) : x 2, the 3 the power costrat says that each codeword x (x,,...,x, ) must satsfy a (x ) a(x k ) PT PT 2BT P : A. 2B Hece, for fxed power costrat P, every operatg pot (Ab,m db (A),) o the parametrc curve correspods to a dfferet badwdth B. I partcular, for large B ( the wdebad regme 5]), the slope of the parametrc curve s well approxmated by (3). I fact, sce the parametrc curve (A),) descrbes the capacty C as a fucto of the (A db b,m 2 I the case of AWG, there s o loss of formato performg ths operato; geeral, we just do t as a practcal sgal processg operato. 3 ote that for a(x) x 2, A m 0. If A 0, the the oly codeword that satsfes a (x) A s the all zeros codeword. Hece, the oly achevable rate wth A 0 s zero; thus, C(0) 0. 8