Revision of Lecture 4

Size: px

Start display at page:

Download "Revision of Lecture 4"

Joshua Simon
6 years ago
Views:

1 Revision of Lecture 4 We have completed studying digital sources from information theory viewpoint We have learnt all fundamental principles for source coding, provided by information theory Practical source coding is guided by information theory, with practical constraints, such as performance and processing complexity/delay trade off When you come to practical source coding part, you can smile as you should know everything Information theory does far more, in fact it provides guiding principle for everything in communications For example, what happens to information transmitted through channel? source (Tx) channel destination (Rx) 55

2 Information Across Channels For conveying information across a transmission channel, a suitable model is: AWGN input channel output The channel itself introduces amplitude and phase distortion, is potentially timevarying, and has a limited bandwidth B Error-free reception of symbols is additionally impeded by additive white Gaussian noise (AWGN); it severeness is described by the signal-to-noise ratio (SNR) Therefore, dependent on the above parameters, we are interested in determining the maximum possible error-free information transmission (channel capacity C) We will see that C depends on B and SNR 56

3 Characteristics of Channel The channel can be described by its impulse response h(t) or equivalently its frequency response H(jω) = A(ω) e jφ(ω) with amplitude response A(ω) and phase response Φ(ω); h(t) and H(jω) are Fourier pair Ideal channel (pure delay): h(t) = δ(t T) A(ω) =, Φ(ω) = ωt A(ω) Φ(ω) B ω B ω T Flat magnitude and linear phase ( = constant group delay G(ω) = Φ(ω)/ ω) The only impairment caused by an ideal channel is AWGN Non-ideal channel: channel is dispersive, causing intersymbol interference 57

4 Additive White Gaussian Noise Noise is uncorrelated with the signal 3 2 time series Gaussian noise has a bell shaped probability density function (normally distributed) p(x) = 2πσ e (x µ)/2σ2 x[n] time index n 3 histogram.4 probability density function with mean µ and variance σ 2 White noise has zero mean, and channel noise is usually modelled as an AWGN counts sample value x p(x) sample value x 58

5 White Noise White noise is characterised by a flat power spectral density function, N(ω), or equivalently, its impulse-shaped auto-correlation function, R(τ) N(ω) R( τ) N 2 N 2 ω τ N(ω) and R(τ) are a Fourier pair: R(τ) = 2π Z N(ω)e jωτ dω N(ω) = Z R(τ)e jωτ dτ Two-side spectrum is usually used for convenience, and N is the noise power 59

6 Physic Basis of Channel Transmitted signal is amplified to required power level and launched from transmit antenna to channel Signal power is attenuated, as it travels in distance path loss Copies of signal arrive at receiver with different attenuation and delays, which may cause dispersive and fading (power level fluctuates rapidly) effects Received signal at receiver is very weak, and needs to be amplified to required power level in order to detect digital information contained While receiver amplifier amplifies receive signal, it also introduces thermal noise AWGN in our channel model in fact models this noise How much noise introduced by amplifier is specified by its noise figure Depending on communication carrier frequency, channel bandwidth and actual communication conditions, channel may be modelled as: AWGN channel, i.e. channel is nondispersive or memoryless Or dispersive channel, i.e. channel has memory 6

7 Binary Symmetric Channel (BSC) BSC is the simplest model for information transmission via a discrete channel (channel is ideal, no amplitude and phase distortion, only distortion is due to AWGN): P(X ) P(X ) X X P(Y X ) P(Y X ) P(Y X ) P(Y X ) Y Y P(X i ): probability of occurrence of X i at source output, a priori probability P(Y j X i ): conditional probability of Rx Y j given Tx X i Symmetric refers to P(Y X ) = P(Y X ) = p e, p e being channel error probability The joint probability P(Y j, X i ) (Tx X i and Rx Y j ) is linked with the conditional probabilities P(Y j X i ) by Bayes rule: P(Y j,x i ) = P(X i ) P(Y j X i ) = P(Y j ) P(X i Y j ) = P(X i,y j ) 6

8 Binary Symmetric Channel Example Consider a BSC: P(X = ) =.7 P(X = ) =.3 p e =.2 TX p e =.98 p e =.2 p e =.98 RX This has a non-equiprobable source with P(X = ) =.7 and P(X = ) =.3: on average, 7% of transmitted bits are and 3% are Channel s error probability p e =.2: on average, bit error rate is 2% 62

9 Binary Symmetric Channel Example (continue) Probability of correct reception: Pcorrect = P(Y =, X = ) + P(Y =, X = ) =.98, as P(Y =, X = ) = P(X = ) P(Y = X = ) =.7.98 =.686 P(Y =, X = ) = P(X = ) P(Y = X = ) =.3.98 =.294 Probability of erroneous reception: Perror = P(Y =, X = ) + P(Y =, X = ) =.2, as P(Y =, X = ) = P(X = ) P(Y = X = ) =.3.2 =.6 P(Y =, X = ) = P(X = ) P(Y = X = ) =.7.2 =.4 Total probability of receiving a (or a ): P(Y = ) = P(X = ) P(Y = X = ) + P(X = ) P(Y = X = ) = =.692 P(Y = ) = P(X = ) P(Y = X = ) + P(X = ) P(Y = X = ) = =.38 63

10 A Close Look at Channel For above (binary symmetric channel) example P(X )=.3 P(Y )=.38 P(X )=.7 P(Y )=.692 Something happens in channel to information How to describe this? Y i and X j are connected They have something in common or mutual P(X ) i X i P(Y X ) i i Y i P(Y ) i P(X ) j X j P(Y X ) j i 64

11 Mutual Information Definition of mutual information of X i and Y j : I(X i,y j ) = log 2 P(X i Y j ) P(X i ) (bits) Perfect, noiseless channel: Y i = X i, i.e. P(X i Y i ) = and I(X i, Y i ) = log 2 P(X i ) This is the information of X i, hence no information is lost in the channel Extremely noisy channel with error probability.5 Y i is independent of X i, hence P(X i Y i ) = P(X i, Y i ) = P(X i) P(Y i ) P(Y i ) P(Y i ) Therefore I(X i, Y i ) = log 2 =, meaning all information is lost in the channel In general, I(X i ) > I(X i, Y i ), some information is lost in the channel 65

12 Mutual Information Example Consider the earlier BSC example: P(X ) =.7 X p e =.98 p e =.2 p e =.2 Y P(X Y ) =.993 P(X Y ) =.87 P(X Y ) =.455 P(X ) =.3 X p e =.98 Y P(X Y ) =.9545 Here, the mutual information results in: I(X, Y ) =.52 bits I(X, Y ) =.67 bits (source info: I(X ) =.55 bits I(X ) =.737 bits) (destin info: I(Y ) =.53 bits I(Y ) =.699 bits) I(X, Y ) = 5.3 bits I(X, Y ) = bits Destination info contents are more balanced then source info contents The negative quantities represent mis-information 66

13 Average Mutual Information Based on received symbols Y j given transmitted symbols X i through a BSC, average mutual information is defined as: I(X,Y ) = i = i P(X i, Y j ) I(X i, Y j ) j j P(X i, Y j ) log 2 P(X i Y j ) P(X i ) (bits/symbol) This gives the average amount of source information acquired per received symbol by the receiver, and should be distinguished form the average source information (entropy H(X)) Note that due to Bayes: P(X i Y j ) P(X i ) = P(X i, Y j ) P(X i ) P(Y j ) = P(Y j X i ) P(Y j ) 67

14 Imperfect Channel: Information Loss Consider re-arranging the mutual information between transmitted symbol X i and received symbol Y j : I(X i,y j ) = log 2 P(X i Y j ) P(X i ) = I(X i ) I(X i Y j ) = log 2 P(X i ) log 2 P(X i Y j ) I(X i,y j ) is the amount of information conveyed to receiver when transmitting X i and receiving Y j, I(X i ) is the source information of X i, and I(X i Y j ) can be regarded as the information loss due to the channel Therefore, I(X i ) I(X }{{} i,y j ) }{{} Source Inf. Inf. conveyed to rec. = I(X i Y j ) }{{} Inf. loss I(X i Y j ) I(X i ), see for example the previous cases of p e = and p e =.5 68

15 Imperfect Channel: Average Mutual Information Average mutual information is given by: I(X, Y ) = X X P(X i Y j ) P(X i, Y j ) log 2 (bits/symbol) P(X i j i ) But this average conveyed information I(X, Y ) = X X P(X i, Y j ) log 2 P(X i j i ) X X P(X i, Y j ) log 2 P(X i j i Y j ) = X P(X i, Y j ) A log 2 P(X i j i ) X! X P(Y j ) P(X i Y j ) log 2 P(X j i i Y j ) = X i P(X i ) log 2 P(X i ) X j I(X, Y ) {z } av. converyed information A similar re-arrangement leads to: I(X, Y ) {z } av. converyed information P(Y j ) I(X Y j ) = H(X) H(X Y ) = H(X) {z } H(X Y ) {z } av. source information av. information lost = H(Y ) H(Y X) {z } {z } destination entropy error entropy 69

16 Summary General consideration for transferring information across channels Channel characteristics or channel model Binary symmetric channel assumptions Mutual information between channel input X(k) {X i } and channel output Y (k) {Y i } characterises how information is transferring across channel Average mutual information I(X,Y ) I(X,Y ) }{{} av. conveyed information I(X,Y ) }{{} av. conveyed information = H(X) H(X Y ) }{{}}{{} source entropy av. information lost = H(Y ) H(Y X) }{{}}{{} destination entropy error entropy 7

Revision of Lecture 5

Revision of Lecture 5 Information transferring across channels Channel characteristics and binary symmetric channel Average mutual information Average mutual information tells us what happens to information