12. Maximum Entropy and Spectrum Estimation

Transcription

1 : Maxu Etropy ad pectru Estato Maru Jutt Overvew Maxu etropy dstrbutos are studed The proble of spectru estato s troduced ad axu etropy based solutos for t are llustrated ource The ateral s aly based o Chapter of the course boo [] Teleco Laboratory Course Overvew Basc cocepts ad tools Itroducto Etropy, relatve etropy ad utual forato 3 Asyptotc equpartto property 4 Etropy rates of a stochastc process ource codg or data copresso 5 Data copresso Chael capacty 8 Chael capacty 9 Dfferetal etropy 0 The Gaussa chael Other applcatos Maxu etropy ad spectral estato 3 Rate dstorto theory 4 Networ forato theory Teleco Laboratory Outle of the Lecture Revew of the last lecture Itroducto Maxu etropy dstrbutos pectru estato Etropy rates of a Gaussa process Burg s axu etropy theore uary Refereces Chael put at te X Revew of Last Lecture The Gaussa chael wth dscrete te dex ad cotuous apltude The ost portat cotuous alphabet chael s the Gaussa chael Wthout costrats o the sgal-to-ose rato (NR) the capacty would be fte Usually a power costrat s appled for a codeword of legth : x P = Gaussa ose at te Z ~N(0,σ Z ); depedet o put X Σ Chael output at te Y = X + Z Teleco Laboratory 3 Teleco Laboratory 4

2 Capacty wth Nyqust Badwdth hao Boud for Codg Methods: Probablty of Error Boud vs NR per Bt The u badwdth for relable (tersybol terferece free) coucato [Dgtal Coucatos] s the Nyqust badwdth: W Nyq = R b /R The capacty per te ut wth P = R b E b becoes P RbE b C = W log + = W log + N 0W N 0W The capacty per sgle chael use wth W = W Nyq = R b /R: P RbE b RE b C = log log log + N0W = + = + N0Rb /R N 0 Teleco Laboratory 5 Teleco Laboratory 6 Requred NR per Bt vs Code Rate Mu NR for Relable Coucatos If tae the ltg case of fte badwdth (W ): RbE b RbE b C = W log + log e, W N 0W N 0 The faous lower boud for NR per bt for relable coucatos: RbE b C = loge Rb N E b N = 59 db loge Teleco Laboratory 7 Teleco Laboratory 8

3 pectral Effcecy Oe ca slarly derve a boud for spectral effcecy r = R b /W [(bts/s)/hz]: E r b N r 0 Teleco Laboratory 9 Parallal Gaussa Chael Model Chael put vector: T [ ] X = X X X Nose vector: T Z = Z Z Z ~ N 0, Chael output vector: [ ] Y = Y Y Y = X + Nose covarace: K = dag σz,, σz,,, σz, Ucorrelated for the te beg The coo power costrat: E X P = K T [ ] Z [ ] Z ~ N (0, σ X Y Z ~ N (0, σ X Parallel Gaussa chaels Teleco Laboratory 0 Σ Σ Z, Z, X Y ) ) Z ~ N (0, σz, ) Σ Y Power Allocato Optzato: Water-Fllg oluto Optal Power Allocato oluto wth Colored Gaussa Nose P = = ( ν σ ) ( ν σ ) + Z, + Z, = P, x, x 0 0, x < 0 + ( x ) =, Power P P P 3 σ Z, σ Z, σ Z,3 P 4 =0 σ Z,4 Ch # Ch # Ch #3 Ch #4 ce for ay atrces B ad C tr(bc) = tr(cb), the trace the power costrat becoes tr(a) = tr(q T K X Q) = tr(qq T K X ) = tr(k X ) Maxze A+Λ subject to tr(a) P Apply Hadaard equalty: K K wth equalty f ad oly f K s dagoal A + Λ A = ( + λ ) larly to the water-fllg, the soluto s + ( ν λ ), A P A = = = Teleco Laboratory Teleco Laboratory

4 Feedbac Capacty The capacty wth feedbac per oe trassso: ( ) C K ax X + Z,FB = log + bts, ( ) ( ) C tr K P K X where the capacty wthout feedbac C s ( ) ( ) ax log KX + KZ C = tr K P K X Ay achevable code satsfes ( ) K X + Z R = log + ε, ε 0, K Z Z Z Itroducto Maxu etropy dstrbutos have bee cosdered therodyacs to characterze the olecule veloctes or crostates gas Maxwell-Boltza dstrbuto Maxu etropy dstrbutos ca be utlzed paraetrc spectru estato ad other slar estato probles elect such a estate value whch assues the least about the data, e, axzes the etropy Robustess Teleco Laboratory 3 Teleco Laboratory 4 Maxu Etropy Dstrbutos Cosder the proble of selectg the dstrbuto so that the dfferetal etropy s axzed Gaussa PDF s the soluto wth covarace costrat Maxze the dfferetal etropy h(f) over all PDF s satsfyg (r(x) s a arbtrary fucto used oet costrat) f(x) 0, wth equalty outsde the support set f ( x ) dx =, f 3 = oluto approaches: calculus forato equalty ( x ) r ( x ) dx α, =,,, Approach #: Calculus Dfferetal etropy s a cocave fucto over a covex set For fuctoal: J ( f ) = f l f + λ 0 f + λ fr = Dfferetate wth respect to f(x), the copoet of f: J = lf ( x ) + λ 0 + λr ( x ) f ( x ) = ettg the dervatve to zero yelds: λ0 + λ r ( x ) = f ( x ) = e, x, where λ 0, λ,, λ are chose to satsfy the costrats Teleco Laboratory 5 Teleco Laboratory 6

5 Approach #: Iforato Iequalty The calculus approach above dd ot prove that the solved PDF yelds deed the axu etropy, but t rather just suggest a approprate for for the correct soluto Iforato equalty ca be used for a rgorous dervato of the axu etropy dstrbuto Ths s foralzed the followg theore Theore: Maxu Etropy Dstrbuto Let where λ 0, λ,, λ, are chose so that f* satsfes f(x) 0, wth equalty outsde the support set 3 λ 0 + λ r ( x ) = f =, x, f x x e = f λ f Fucto f* uquely axzes h(f) over all PDF s satsfyg the costrats The sae approach ad slar soluto hold also for dscrete etropes, ad ultvarate dstrbutos ( x ) dx =, ( x ) r ( x ) dx = α, =,,, Teleco Laboratory 7 Teleco Laboratory 8 Proof Let g be ay PDF whch satsfes the costrats 3 above g h ( g ) = g lg = g l f = D ( g f ) g lf f Equalty f ad oly f g(x) = f*(x) g lf = g l exp λ 0 + λr ( x ) = = f λ 0 + λr ( x ) = = f l f = h f by o-egatvty of relatve etropy by defto of f* both f* ad g satsfy the costrats Teleco Laboratory 9 Exaple #: Frst ad ecod Moet Costrats Cosder the case = (, ) = R Let the costrats be E(X) = 0 ad E(X ) = σ Falar varace costrat The soluto s the Gaussa (oral) dstrbuto N(0,σ ) To verfy t, let us ote that E(X) = 0 ad E(X ) = σ E ( X ) = 0 xf ( x ) dx = 0, Accordg to the prevous theore: ( X ) E = σ x f x dx = σ f x = e λ 0 +λ x +λ x Teleco Laboratory 0

6 Exaple #: No Costrats ad Closed Iterval Cosder the case = [a,b] Assue o costrats except the trval frst two b oes: f ( x ) dx =, The soluto s of the for Ufor dstrbuto a λ = [ ] f x e 0, x a, b Exaple #3: No Costrats vs the Frst ad ecod Moet Costrats Cosder the case = (, ) = R Assue o costrats The axu etropy s fte ad there s o axu etropy PDF Cosder, eg, Gaussa PDF wth σ Let the costrats be geeral frst ad secod oet costrats E(X) = µ ad E(X ) = α The soluto s the Gaussa (oral) dstrbuto N(µ,α µ ) Follows drectly fro the fact that Gaussa PDF axzes the dfferetal etropy uder covarace costrat Teleco Laboratory Teleco Laboratory Exaple #4: Multvarate ecod Moet Costrats Cosder the case = R wth the secod oet costrats E(X X ) j = K j λ 0 + λj x x j, j = The axu etropy PDF: f ( x) = e Aga, sce the expoet s of a quadratc for, t s clear by the earler results that the soluto ust be the ultvarate Gaussa PDF: T f ( x) = f ( x, x,, x ) = exp, x K x K ( π) h ( f ) = h ( X, X,, X ) = log ( πe ) K Teleco Laboratory 3 Exaple #5: A Aoalous Maxu Etropy Proble () Let us cosder a slar set-up as Exaple #: Now we add a thrd costrat there: The soluto ust be of the for: f x = e However, f λ 3 0 f x dx = The PDF caot be oralzed correctly xf x dx = α xf x dx = α 3 λ 0 +λ x +λ x +λ3 x, Teleco Laboratory 4 3 x f x dx = α 3

7 Exaple #5: A Aoalous Maxu Etropy Proble () Four equatos ad three varables I geeral, the soluto does ot exst The dfferetal etropy has a upper boud gve by the Gaussa PDF, but t s ot possble to atta t However, we ca get arbtrarly close Cocluso: suph(f) = h(n(µ,α α )) = ½lπe(α α ) Exaple #6: Dce Cosder dce [oppa, Fsh] so that = {,,3,4,5,6} Dscrete RV dscrete etropy s cosdered Wth far dce wth p(x) = /6, the etropy s axzed wth the ufor dstrbuto Wth a costrat E(X) = α, [Boltza] the proble becoes ore trcy λ e The soluto s [, Exaple 3]: p =, 6 λ where λ s such that 6 e p = α = = Teleco Laboratory 5 Teleco Laboratory 6 pectru Estato Autocorrelato fucto of a statoary stochastc process: R ( ) = E( X X + ) Power spectral desty (Fourer trasfor): jλ λ = R e, π λ π = pectru estato attepts to estate the power spectral desty or equvaletly the autocorrelato everal ethods exst A deeper treatet the course Coucato gal Processg I pectru Estato Methods Proble: Fro a fte-legth data record, detere a estate of the power spectral desty Noparaetrc ethods: No paraetrc odel of the syste assued Methods: perodogra-based ethods ultple wdow ethod Paraetrc ethods: Assue a paraetrc stochastc odel Methods: odel detfcato procedures u varace dstortoless respose ethod egedecoposto-based ethods Teleco Laboratory 7 Teleco Laboratory 8

8 Paraetrc Methods Assue a paraetrc stochastc odel Model detfcato procedures Mu varace dstortoless respose ethod 3 Egedecoposto-based ethods Noparaetrc Methods No assuptos tartg pot: the fudaetal equato Two dfferet approaches ca be dstgushed Perodogra-based ethods perodogra U N (ω) /N s used as the startg pot Multple wdow ethod the fudaetal equato appled pectra coputed usg the paraetrc ethods ted to have sharper peas ad hgher resoluto tha those obtaed fro the oparaetrc ethods Teleco Laboratory 9 Teleco Laboratory 30 Perodogra Method = aple estate: Rˆ = X X + splest Perodogra ethod: estate the spectru based o saple estates More saple values for low lags (sall values of ) tha hgh lags (large values of ) If correlato coverges slowly to zero, the spectru estate wll be based Poor estates et large lag autocorrelatos to zero, use wdowg Maxu Etropy Method The axu etropy ethod (MEM) or Burg s ethod s based o forato theoretc arguets To ze the ucertaty related to the easureets, choose the ost upredctable estate, e, the oe whch axzes the etropy (or ucertaty) of the uderlyg rado varable Bypasses the probles of wdowg fuctos perodc exteso of the data or assupto of zero data outsde the observato terval Noegatve at all frequeces Teleco Laboratory 3 Teleco Laboratory 3

9 Etropy Rates of a Gaussa Process Etropy rate of a dscrete valued stochastc process was studed earler It ca be exteded for a cotuous-valued stochastc processes as well Dfferetal etropy rate (f the lt exsts): h ( X, X,, X ) h ( X) = l For statoary processes: h( X) = l h( X X,, X, X) For a statoary Gaussa stochastc process: ( ) h ( X, X,, X ) = log ( πe ) det( K ), where K () s covarace atrx wth K () j =R( -j ) Etropy Rate of a Gaussa Process () As, the egevalues of the covarace atrx approach the spectru of the stochastc process Etropy rate of a statoary Gaussa stochastc process: π hx = log( πe ) + log[ ( λ) ] dλ 4π π Varace of the best estator gve the fte past: h( X σ ) = πe Teleco Laboratory 33 Teleco Laboratory 34 Burg s Maxu Etropy Theore The axu etropy rate stochastc process {X} satsfyg E X X = α, = 0,,, p +, s the pth order Gauss-Marov process: p X = a X + Z, = where Z are IID zero ea Gaussa RV s wth varace σ Costats are chose to satsfy the costrats, ad ca be solved by Yule-Waler equatos Rears Rear: {X } s ot assued to a) have zero ea b) be Gaussa c) Be statoary Proof: ee textboo [, ect 6] for coplete proof The dfferetal etropy of a fte seget of a stochastc s bouded above by the etropy of a seget of a Gaussa rado process wth the sae covarace structure Ths s tur bouded above by the al order Gauss- Marov process Characterzed by Yule-Waler equatos Teleco Laboratory 35 Teleco Laboratory 36

10 Yule-Waler Equatos ad the Maxu Etropy pectru Estate The pth order Gauss-Marov process s characterzed by the well-ow Yule-Waler p equatos: R ( 0 ) = a R ( ) + σ, p+ equatos ad uows a uque soluto exsts Fast soluto by Levso algorth The spectru of the axu etropy process: σ ( l ) = p jl + a e = p R l = a R l, l =,, = = Teleco Laboratory 37 uary Maxu etropy dstrbuto: Maxu etropy spectru estate: λ 0 + λ r ( x ) = = =,, f x f x e x λ ( l ) = p = jl The paraeters are solved fro Yule-Waler equatos Ca be used other probles as well Exaple: rado chael odelg σ + a e Teleco Laboratory 38 Refereces o Hay, Adaptve Flter Theory, 3rd ed Pretce Hall, 996 IBN: X P toca & R Moses, Itroducto to pectral Aalyss Pretce-Hall, 997 IBN M Jutt, Coucato Processg I Lecture otes, Uversty of Oulu, Dept Electrcal ad Ifor Eg, Teleco Laboratory Teleco Laboratory 39