Basics of Information Theory: Markku Juntti. Basic concepts and tools 1 Introduction 2 Entropy, relative entropy and mutual information

Transcription

1 : Maru Jutt Overvew he propertes of adlmted Gaussa chaels are further studed, parallel Gaussa chaels ad Gaussa chaels wth feedac are solved. Source he materal s maly ased o Sectos.4.6 of the course oo [] ad also Beedetto & Bgler [3, Sect. 3.3]. elecomm. Laoratory Course Overvew Basc cocepts ad tools Itroducto Etropy, relatve etropy ad mutual formato 3 Asymptotc equpartto t property 4 Etropy rates of a stochastc process Source codg or data compresso 5 Data compresso Chael capacty 8 Chael capacty 9 Dfferetal etropy he Gaussa chael Other applcatos Maxmum etropy ad spectral estmato 3 Rate dstorto theory 4 Networ formato theory elecomm. Laoratory Outle of the Lecture Revew of the last lecture Commucato theoretc ouds arallel Gaussa chaels Colored Gaussa ose chaels Gaussa chaels wth feedac Summary Revew of Last Lecture Gaussa ose at tme ~N(, ); Chael put at tme depedet o put Chael output at tme Y = + he Gaussa chael wth dscrete tme dex ad cotuous ampltude. he most mportat cotuous alphaet chael s the Gaussa chael. Wthout costrats o the sgal-to-ose rato (SNR) the capacty would e fte. Usually a power costrat s appled for a codeword of flegth : x. elecomm. Laoratory 3 elecomm. Laoratory 4

2 Capacty of Gaussa Chael he (operatoal) capacty of a Gaussa chael wth power costrat ad ose varace s log. C I other words, all rates elow chael capacty C are achevale. For all rate R < C: a sequece of ( R,) codes so that, as. Coverse: For ay sequece of achevale ( R,) codes R C. (t) Bad-Lmted Chael Model (t) h(t) Y(t) W H(f) Assume a deal adpass chael mpulse respose h(t). Chael output: Y t t t h t h t t h t t. I other words, oth the ecoded formato sgal ad ose are ad-lmted. W elecomm. Laoratory 5 elecomm. Laoratory 6 Capacty of Bad-Lmted Gaussa Chael Remd the capacty of Gaussa chael per oe sgle sample trasmsso: log. C Capacty per sample: C log. N W Capacty per tme ut ((W) samples per secod): C W log. N W Commucato heoretc Bouds Assume that we have ary data to e trasmtted over a adlmted Gaussa chael. Called formato data symols. Assume that the formato t rate [ts/s] of the source s R. he eergy e per formato o t s E = /R/ or = R E. he formato data s the ecoded y a forward error cotrol (FEC) code wth rate R = / to yeld chael data ts trasmtted over a commucato chael. he ecoded t rate trasmtted to the chael s R /R, where R s the code rate. elecomm. Laoratory 7 elecomm. Laoratory 8

3 Capacty wth Nyqust Badwdth he mmum adwdth for relale (tersymol terferece free) commucato [Dgtal Commucatos] s the Nyqust adwdth: W Nyq = R /R. he capacty per tme ut wth = R E ecomes RE C W log W log. N W N W he capacty per sgle chael use wth W = W Nyq = R /R: RE RE C log log log. NW NR /R N Rate ad SNR Bouds Fao s equalty for BSC: H Y H e e log. It ca e show (for detals, see [3, Sect. 3.3]) that the code rate must satsfy C R H. e Applyg ths to the adlmted Gaussa chael, we get RE log R N E R. H e N R By settg equalty, curves ca e draw. elecomm. Laoratory 9 elecomm. Laoratory roalty of Error Boud vs. SNR per Bt Requred SNR per Bt vs. Code Rate elecomm. Laoratory elecomm. Laoratory

4 Mmum SNR for Relale Commucatos If tae the lmtg case of fte adwdth (W ): R E R E C W log log e, W. N W N hefamousloweroudforsnrpertforrelale for t for relale commucatos: R E C loge R N E db. N loge Spectral Effcecy Oe ca smlarly derve a oud for spectral effcecy r = R /W [(ts/s)/hz]: r E. N r elecomm. Laoratory 3 elecomm. Laoratory 4 arallel Gaussa Chaels Cosder depedet parallel Gaussa chaels (or suchaels) wth commo power costrat jot ecodg ad power allocato jot decodg depedet addtve Gaussa ose each chael geeralzed soo for correlated ose case. I other words, we do ot have dfferet commucato prolems, ut parallel chaels to carry the same message. elecomm. Laoratory 5 arallal Gaussa Chael Model ~ N (,, ). Chael put vector: Y. Nose vector: ~ N (,, ). Y ~ N,. Chael output vector: Y Y Y Y. Nose covarace: ~ N (,, ). dag,,,,,,. Y Ucorrelated for the tme eg. he commo power costrat: arallel Gaussa chaels. E. elecomm. Laoratory 6

5 Mutual Iformato Boud for arallel Gaussa Chaels Smlarly as sgle chael case, the mutual formato ecomes I ; Y h Y h Y h Y h h Y h h Y h. Sce the ose compoets are depedet, we get I ; Y h Y h log e log e,,,. log, E, elecomm. Laoratory 7 Capacty of arallel Gaussa Chaels Equalty ca e acheved the mutual formato oud f N, dag,,,. ~ he capacty of parallel Gaussa chaels: C max I f x log, ; Y. he remag prolem: how to allocated the powers to the suchaels so that the aove maxmum s acheved? Waterfllg. elecomm. Laoratory 8 ower Allocato Optmzato : rolem Set-Up Optmzato prolem: Maxmze suject to C log. Lagrage fuctoal:, J,,,, log.. ower Allocato Optmzato: Dfferetato Dfferetate the Lagrage fuctoal ad set dervatve to zero: J,,,,. Aove s selected so that the power costrat s satsfed,.e.,. Sce the power caot e egatve the fal soluto s foud y uh-ucer codtos., elecomm. Laoratory 9 elecomm. Laoratory

6 ower Allocato Optmzato: Water-Fllg Soluto Applcatos of arallel Chaels,,,, x, x, x x. ower 4 = 3,,, 3, 4 Ch # Ch # Ch #3 Ch #4 Suchaels multcarrer commucatos. ower ad data rate adapto (ofte called t loadg) over sucarrers. Spatal chaels multatea multple-put multple-output (MIMO) commucatos. Smlar power, data ad phase adaptato possle. I wreless commucatos, the parallel chaels ca e ether for creased relalty y dversty same data over several parallel suchaels creased data rate y multplexg dfferet data over dfferet suchaels. elecomm. Laoratory elecomm. Laoratory Colored Gaussa Nose Chaels We have so far assumed that the ose s whte (ucorrelated) oth temporally (adlmted Gaussa chael) or over suchaels (parallel Gaussa chaels). he treatmet ca e geeralzed to colored (correlated) Gaussa ose for oth cases. emporal correlato (ose wth memory) s modelled wth a loc of cosequtve uses of the chael. he parallel chael case s modelled wth parallel depedet ose processes. he same mathematcal treatmet for oth cases. ad are the covarace matrces of the ose ad the put, respectvely. Mutual Iformato ower costrat: E tr, where tr() deotes the trace of a matrx,.e., the sum of the dagoal elemets. Note that ow the costrat depeds o. Mutual formato s le wth whte ose, sce the ose ad put are stll depedet from each other: I ; Y h Y h Y h Y h h Y h h Y h. he aove s mazmzed whe Y s Gaussa whch requres to e Gaussa. elecomm. Laoratory 3 elecomm. Laoratory 4

7 Dfferetal Etropy ad Egevalue Decomposto he covarace of the output Y: Y = +. he dfferetal etropy: h Y h Y, Y,, Y log e Y log e. Apply the egevalue decomposto = QQ Q, where Q s orthogoal,.e., QQ = Q Q = I, ad s a dagoal matrx wth egevalues of at the dagoal. QΛQ Q Q Q Λ Q Q Q A Λ, Q Λ Q A Q Q Q. Q Λ elecomm. Laoratory 5 Optmal ower Allocato Soluto Sce for ay matrces B ad C tr(bc) =tr(cb) tr(cb), the trace the power costrat ecomes tr(a) = tr(q = Q) tr(qq ) = tr( ). Maxmze A+ suject to tr(a). Apply Hadamard a d equalty: wth equalty f ad oly f s dagoal. A Λ A A. Smlarly to the water-fllg, the soluto s A, A. elecomm. Laoratory 6 Chaels wth Memory: Spectral Waterfllg Cosder a chael wth colored ose,.e., ose wth memory or o-uform power spectral desty (SD). If the process s statoary, the covarace matrx s oepltz, the egevalues have a lmt as, ad ther desty teds to the SD of the ose process. Water-fllg the spectral doma. Gaussa Chaels wth Feedac W ~ N (, ). Y = + he Gaussa chael wth feedac. We showed that feedac does ot crease the capacty of memoryless dscrete chaels. he same s true for memoryless Gaussa chaels. For chaels wth memory (correlated ose from tme stat to aother), capacty ca e creased. elecomm. Laoratory 7 elecomm. Laoratory 8

8 rolem Defto: Feedac Codes he chael outputs are assumed to e avalale at the trasmtter. A ( R,) feedac code s a sequece of mappgs x (W,Y ), where each symol x s a fucto oly of the message W ad prevous receved values Y,Y,,Y. I addto, we have the power costrat w, Y, w,,,. R x Because of feedac, ad deped o each other. ey Results o Feedac Capacty he capacty wth feedac per oe trasmsso: C,FB max log C ts, tr where the capacty wthout feedac C s max C log. tr Ay achevale code satsfes R log,,. elecomm. Laoratory 9 elecomm. Laoratory 3,,,, x, x, x x. Summary: Water-Fllg Soluto ower 3 4 =,,, 3, 4 Ch # Ch # Ch #3 Ch #4 ey Results o Feedac Capacty he capacty wth feedac per oe trasmsso: C,FB max log C ts, tr where the capacty wthout feedac C s C max log. tr Ay achevale code satsfes R log,,. elecomm. Laoratory 3 elecomm. Laoratory 3