Channel Coding 1. Sportturm (SpT), Room: C3165

Size: px

Start display at page:

Download "Channel Coding 1. Sportturm (SpT), Room: C3165"

Moris Houston
5 years ago
Views:

1 Channel Coding Dr.-Ing. Dirk Wübben Institute for Telecommunications and High-Frequency Techniques Department of Communications Engineering Room: N3, Phone: 4/ Sportturm (SpT), Room: C365 Lecture Monday, 8:3 : in S7 Eercise Wednesday, 5: 7: in N5 Dates for eercises will be announced during lectures. Tutor Shayan Hassanpour Sportturm Room: (SpT), N39 Room: C3 Phone hassanpour@ant.uni-bremen.de

2 Outline Channel Coding I. Introduction Declarations and definitions, general principle of channel coding Structure of digital communication systems. Introduction to Information Theory obabilities, measure of information SHANNON s channel capacity for different channels 3. Linear Block Codes operties of block codes and general decoding principles Bounds on error rate performance Representation of block codes with generator and parity check matrices Cyclic block codes (CRC-Code, Reed-Solomon and BCH codes) 4. Convolutional Codes Structure, algebraic and graphical presentation Distance properties and error rate performance Optimal decoding with Viterbi algorithm

3 Definitions Chapter. Information Theory Measure of information, Entropy Entropies of a communication system SHANNON s channel capacity for different channels Channel with discrete output alphabet Channel with continuous output alphabet Capacity for continuous input alphabet Capacity of bandlimited channel Gallager Eponent and Cut-off Rate Bhattacharyya bound for the error probability Gallager function and Gallager eponent Cut-Off Rate Appendi 3

4 DEFINITIONS 4

5 Basics of Information Theory Basic for the conception of all communication systems, established by C.E. Shannon in 948 Basic question: What is the maimum rate that can be transmitted over a given channel without errors? This maimum rate is known as channel capacity Let be a random variable taking values in the alphabet,,, with probability How to measure the amount of information of a symbol Amount of information should be non-negative and real: Amount of information should depend on probability: I X f X For independent events the common information corresponds to the sum of the individual contents: X Y X Y I X Y I X I Y,, I X I X Logarithm is sole function that maps product onto a sum, 5

6 Amount of information per digit: I X log log X Information, Entropy log bits (binary digits) log e nats (natural digits), log hartley. Symbols with small probability yield large. amount of information, whereas likely symbols contain only small amount of information {X } Entropy: Average amount of information of a symbol alphabet,, X E log X X log X X Average amount of information provided by an observation of X Our uncertainty about The randomness of E: appendi X log independent events Convention: log since lim log Note, the entropy is not a function of the random variable, but rather of the set and its probability mass function. 6

7 Entropy (Entropie) For a discrete set of elements the entropy fulfills oof: iff (if and only if) for one for all and log log for all log Using Jensen s inequality for concave function log u u Elog log E X log M M X X log log X log M X X Entropy is maimized, when all M elements are equally likely, i.e., : M ma X equal X log M M log M log M bit X M M Uniform distribution leads to maimum uncertainty most random variable M Jensen inequality for concave f() E E Rules for logarithm in appendi 7

8 Eample: Seven Segment Display c digit a a d f b c d b e g e f g All digits with same probability:. Amount of information per digit: log log 3.3 bit Entropy of alphabet: 3.3 bit Absolute redundancy: 7 bit 3.3 bit 3.68 bit Relative redundancy: / 3.68 bit / 7 bit 5.54% 8

9 Eample: Entropy of a binary Alphabet Given:, with and Entropy X E log X Binary entropy function Maimum entropy of a binary alphabet for equally likely symbols and.5 ma X p log ( p ) ( p ) log ( p ) ( p ) X X bit equal log bit (p) (p ma ) = p =. p =.89 p =.5 ma p 9

10 Given: Eample: Non-uniform Symbol Alphabet X X X X X 3 X 4 X 5 X 6 X 7 {X } / /4 /8 /6 /64 /64 /64 /64 bin. represent Entropy describes the minimum average number of bits to represent all alphabet symbols uniquely (entropy coding / source coding) X log log log log 4 log bits Binary representation: {,,,,,,, } Average description length 4 l E l l X bits Source coding / compression coding (e.g. Huffman coding) In contrast, for uniform probability 3 bits are required for representation!

11 Generation of optimal prefi-free code X / X /4 X /8 X 3 /6 X 4 /64 X 5 /64 X 6 /64 X 7 /64 {A } = {A 6 }+{A 7 } = /3 Huffman Coding X {X } Binary representation /4 {A 3 }={A }+{A } =/6 {A } = {A 4 }+{A 5 } = /3 /8 / while {A i } arrange symbols with decreasing probability Combine variable with lowest prob. to auiliary variable end A i = A a & A b distinguish A a, A b by, {A i }= {A a }+{A b }

12 Illumination of Entropies (X) (Y) (X Y) (X;Y) (Y X) (X,Y) Venn diagram : entropy of source alphabet : entropy of sink alphabet, : joint entropy of source and sink : equivocation: information lost during transmission : irrelevance: information originating not from source ; : mutual information: information correctly sent from source to sink

13 Joint Entropy Joint Entropy (Verbundentropie) X, Y E log X, Y X, Y log X, Y Entropy of random joint variable (, ) uncertainty of observing and jointly X, Y X Y X Y X Y X Y XY XY, E log XY, E log X YX XY,, X Y X X, Y E log log XY, E log X E log YX X YX X, E XY, log X Y Y X Y Y X Y Epectation variable is dropped subseq.: E E Conditioning reduces uncertainty: (Information can t hurt on the average) X Y X YXY 3

14 Chain Rule of Uncertainty Joint Entropy of random variables,,, : Using chain rule of probabilities yields X, X,, X E log X, X,, X N X X XN X X X XN X X XN,,,,,, N n Xn X, X,, X N n X, X,, X N E log X n X, X,, X n N N n N X n X X X n X n X X X n E log,,,,,, n n e.g. X, Y, Z X Y X Z X, Y X Z X Y X, Z 4

15 Equivocation Equivocation (Äquivokation) XY, X Y EXY, log X Y E XY, log Y X, Y Y X Y X Y, log Conditional entropy: Uncertainty about, once is known Information that contains if is known information lost during transmission Derivation : conditional entropy of given the equivocation becomes XYY X Y log X Y XY Y XY Y X log Y Y X Y X og Y r X Y, l P using Y X Y X Y, 5

16 Derivation : XY XY, Y Conditioning reduces uncertainty X Y X Equivocation X, Ylog X, YY logy X Y X Y X, Y log Y, log, X, Y If and are not independent, the knowledge of reduces the randomness about on average Communication: if r signal is given, uncertainty about t signal is reduced og Y X, Y log X, Y l X Y using Y X Y, 6

17 Irrelevance and Mutual Information Irrelevance (Fehlinformation) Mutual Information (Wechselseitige Information, Transinformation) represents uncertainty about before we know, represents the uncertainty about after is received = amount of information provided about by (communication) with operties YX E log YX XY, X X, Y log Y X YX Y XY ; X XY Y Y X X Y X, Y Information that contains if is known e.g. noise X ; Y Y ; X mutuality X Y X, Y Y YX XY, X XY ; 7

18 Mutual Information Mutual Information with Entropy definitions X ; YXlogX Y logy X, Ylog X, Y X, Y X Y X Y Y X Y X X Y Y X Y X X g X, Y, log log lo XY Y X X ; log Y X Y X X X ; Y X Y X, Y a a b, ab ab b, Depends only on transition probabilities and input statistic 8

19 Alternative Forms Alternative Forms for the Mutual Information XY X Y X, Y X Y Y X Y ;, log X Mutual Information is given by the average (taken over and ): You may also find ; in the literature X, Y log X, Y log X YX XY XY, XY ; E log E log E log Y X X Y Y 9

20 Channel Capacity (Kanalkapazität) (X Y) (X) (X;Y) (Y) (Y X) Supremum: least upper bound, e.g., sup / for Channel capacity: Maimum of mutual information over all possible input statistics C sup ( X ; Y ) sup Y X X log { X } { X } Y X Y X X bits per channel use bits/s/hz

21 Noisy-Channel Coding Theorem For every discrete memoryless channel, the channel capacity has the following property: C sup ( X ; Y ) { X } For any and, for large enough, there eists a code of length and rate and a decoding algorithm, such that the maimal probability of block error is. If a probability of bit error is acceptable, rates up to are achievable, where C R c P b P For any, rates greater than are not achievable. In practice optimization of input statistic is often not possible k XY Y X b ( ; ) log k Y Y X X For equally distributed input symbols the mutual information depends only on transition probabilities

22 SHANNON S CHANNEL CAPACITY FOR DIFFERENT CHANNELS

23 Channel Capacity for BSC and Mutual Information for Different Statistics of Input Signal MI(P e ) Mutual Information (MI) of BSC Symmetric input P e {X } =. {X } =.3 {X } =.5 -P e P e X -P e Symmetric input statistic: C ma = bit/s/hz for P e = and P e = error free transmission without coding Capacity decreases with increasing P e C min = bit/s/hz for P e =.5 For P e. C =.5 bit/s/hz with optimal channel coding of rate R c < / error free transmission is possible in theory Non symmetric input statistic: Reduction in mutual information due to symmetry of channel P e Y BSC C P e P e P e P e H P e X log log ( ) Y derivation: appendi & eercise 3

24 Quantization Bounds for BSEC Parameter has to be optimized with respect to channel capacity Optimal choice depends on signal-to-noise-ratio / X -P e -P q Y P q P e X P e P q Y Y -P e -P q X = - X = + -a +a Y Y Y Choice of parameter yields and (by integration of pdf over decision interval) 4

25 Channel Capacity for BSC and BSEC BSEC C P q P e log P e P e P q log P e P q P q log P q BSEC.8 C C.6.4 BSEC, a = opt. BSC a opt. a - - E s /N in db a > leads only to minor improvement of channel capacity 5

26 Channels with Continuous Output Alphabet For AWGN channel the output alphabet is given by Continuous amplitudes for result in integral for capacity calculation For equally likely input symbols : p y X ( ; ) log d k p X k XY p y X In general a quantization with respect to q bits takes place Results in a finite number of output symbols y 6

27 Channel Capacity of BPSK and AWGN Channel for Different Quantization Levels Quantization leads to loss of information C decreases Quantization with q = 3 bits leads only to negligible loss in comparison to q = (no quantization).8.6 C.4 q = q =. q = q = 3 gauss N in db E S / 7

28 Differential Entropy Generalization to continuous random variables X with pdf Differential entropy (Differentielle Entropie) X E log p p log p d (X) does not give the amount of information in X theoretically, infinite number of bits required to represent continuous signal (X) can also be negative no physical interpretation Eample: Uniform distribution / a a a p else For a <.5, log a < (X) is negative With /3: X log 4 a log a X log d log a a a a For a finite domain (i.e. fied maimum absolute value), the uniformly distributed random variable leads to the largest differential entropy. 8

29 Differential Entropy Eample: Normal Distribution, with mean and variance p ( ) ep X log p d log log e ln e d p p log l p d og e p d log lo g X log e e (X) depends only on spread of the distribution (i.e. ), but not on the mean The Gaussian random variable has the largest differential entropy among all continuous random variables of same variance. 9

30 Shaping Gain How much power reduction is possible using Gaussian instead of uniformly distributed random variables of equal differential entropy? Entropy of Uniform distribution with variance log G log X X e U e U.4.53dB 6 G U Entropy of Gaussian distribution with variance G A Gaussian random variable achieves the same differential entropy with an average power.53 db less than required by a uniformly distributed random variable 3

31 Channel Capacity for Continuous Input Alphabet Generalization to models with continuous input alphabets Channel capacity C sup Y Y X p Worst case: Gaussian distributed noise maimizes irrelevance (Y X) To maimize sink entropy (Y) the receive signal should be Gaussian distributed Thus, the transmit signal X needs to be Gaussian distributed as well AWGN-channel with power spectral density / with ~, y~, with y E s N / C Y N log e y log e n log log n N / C log E s N Capacity increases with SNR (channel quality) plot C(E s /N ) on slide 7. 3

32 Channel Capacity for Continuous Input Alphabet Recall: Information vector u is mapped to code vector c of length n>k Energy u k channel encoder E b : Average energy per information bit E s : Average energy of each transmit symbol ( code bit ) c n For fair comparison of coded and uncoded systems, the encoder should not increase the energy (i.e. act as an amplifier) No energy increase due to coding E s < E b k E b n E s E s E b R c E b k n R c k E s n E b 3

33 Channel Capacity for Continuous Input Alphabet Capacity of -D AWGN channel E s E b C log log R c N N R c = C implicit equation E log b C C N SNR required to achieve rate C C E b N C Minimum SNR for information bits C E b ln() lim lim ln().59 db C N C No error free communication possible for E b /N < -.59 db C Eb / N in db -.59 db 33

34 Channel Capacity How does the performance of the system depend on the basic resources? Key observation: f() = log (+) is concave, i.e., f () The higher the SNR, the smaller the effect on capacity log log e for log log for Small SNR: doubling power doubles C High SNR: doubling power C increases by bit C E s /N 34

35 BI-AWGNC Direct calculation using relation from Channels with Continuous.5 log Output Alphabet by numerical integration using the pdfs ep Alternative calculation Differential entropy of noise Sink entropy approimated by Monte-Carlo integration with sufficiently large # trials log log log 35

36 BI-AWGNC C Shannon capacity (Gaussian input/output) Soft decision BI-AWGNC Hard decision BSC C Shannon capacity Hard decision BSC Soft decision BI-AWGNC E s /N Hard-decision prior to decoding results in a loss of to db At / dbthe Shannon capacity is E b /N 36

37 Channel with bandwidth Capacity of a Bandlimited Channel Due to the. Nyquist criterion symbols can be transmitted in time Signal to Noise ratio (bandwidth / ) E s S / N BEs Es / T s n BN N Ts N C BC B log S N C B log S 3.3 log S B log N N n bits/s Eample: Telephone channel with B = 3 khz and S/N = 3 db 3 / C 3Hz log 9,9kbit / s V-9 modem: Requires more bandwidth and higher SNR, e.g. B = 4 khz and S/N = 4 db 4 / C 4Hz log 55,8kbit / s In both directions Only in downlink (digital channel from server to telephone switch) 37

38 Data Rates of Communication Systems over Copper Telephone Lines Transmission System Bandwidth Data rate Analog telephone (POTS) ISDN ADSL (ADSL-over-ISDN, Anne B) ADSL+ (ADSL-over-ISDN) 3 Hz 3.4 khz Hz khz U: 38 khz 76 khz D: 76 khz. MHz U: 38 khz 76 khz D: 76 khz. MHz up to 56 kbit/s (typically 4,5 kbyte/s 5 kbyte/s) 64 kbit/s data channel + 6 kbit/s control channel Upstream: Mbit/s Downstream: up to Mbit/s, Upstream: Mbit/s Downstream: up to 4 Mbit/s, Actual data rate depends strongly on distance to telephone switch 38

39 GALLAGER EXPONENT AND CUT-OFF RATE 39

Channel capacity Gallager Eponent and Cut-Off Rate No statement about the structure of the code or its performance Describes only the asymptotic behavior for very long codes not suitable to estimate

40 Channel capacity Gallager Eponent and Cut-Off Rate No statement about the structure of the code or its performance Describes only the asymptotic behavior for very long codes not suitable to estimate the word error rate given a specific code word length Only a theoretical bound Gallager eponent Achieves a statement for a given code word length n Steps of subsequent derivation Bhattacharyya bound for the error probability Gallager function and Gallager error eponent Cut-Off Rate Interpretation Robert G. Gallager 4

41 Bhattacharyya Bound () Code of rate /with encoding function and decoding function k elements, y n k elements k elements Decoding area: set of receive vectors with decoding result ( i ) i y g y u Decoding areas are disjoint i j for i j 4

42 Bhattacharyya Bound () Error probability for specific information word () i () i () i () i u u u u y u u y P wi ˆ, i () i () i with g u y, is given by sum of transition probabilities over all, i.e. all receive words that lead to decoding error In general, summing over all elements is complicated (description of decoding area) it would be easier to sum up over all possible trick by scaling factor i Special eample with code words (i.e. k = ) P w, y y () Introducing scaling factor () y y () and y y P w, y y () 4

43 Bhattacharyya Bound (3) Multiplying P w, with scaling factor yields upper bound P w Sum over all y achieves further upper bound Decoding areas have not to be known P () () w, y y Equivalent epression for P w, () () y () y r y y () y (), P y y Average word error probability for two code words ({ () }+{ () }=) y () () () () () () y y y ry y y P y y y y () y () P w, () () y () y () P P P w w, w, y y Epression is nonnegative for all y 43

44 Bhattacharyya Bound (4) For a DMC (discrete memoryless channel) n () i () i y y j j j the epression for error probability follows n n () () P y y w j j j j y y j n j y () () y j y j word error probability for a code containing two code words (k = ) Bhattacharyya Bound: Error probability for a general code with k code words k n of length n and ML decoding () i ( ) P y y wi, j j j y i may become very loose if k is large improvement by Gallager 44

45 Gallager Eponent () Including a weighting into the epression for the word error probability () i () i () i () i u u u u y u u y P wi ˆ, i If is transmitted, in case of an error ( ) at least one l fulfills ( ) ( i y y ) ( ) y () i y As this quotient is non-negative for all (), the sum over all quotients k is larger than one. Thus the following relation holds k i with s ( ) y () i y s, y i Aim: choose parameters s and so that the sum tightly fulfills the inequality s < : reduce quotients much larger than < : reduce the sum for k» (a lot of terms in the sum) Gallager factor 45

46 Gallager Eponent () Including the weighting factor in the epression for the word error probability With the choice of and summation over all y the inequality is and simplifies for a DMC to s k ( ) k s s () i () ( ), y i wi y () i i y y y y y i i i P P wi, y s k () i ( ) y y i k n () i ( ) P wi, y j y j j y i Gallager bound: corresponds to Bhattacharyya bound for = upper bound for P w,i Again: due to compleity hard to evaluate for specific code and channel approimation for the error probability by an epectation over all codes 46

47 Gallager Eponent (3) Epectation for P w over all codes under condition () i k P w E Pw, i E \ P, w i y y For a DMC the factorization yields k k with Gallager function n, k E X n n P w y y E, X E,Xlog y y Weighting factor yields an upper bound for word error probability to achieve an appropriate estimation, the inequality has to be tightened with respect to minimize the quotient (Gallager factor) or maimize E (, {X}) Independent of with n j j 47

48 Gallager Eponent and Cut-Off Rate Maimization gives Gallager Eponent E ( R ) ma ma E, X R ma ma log y R X X y each pair of {X} and corresponds to a line with slope - G c c c Up to R crit = achieves maimum Cut-Off Rate ( = Bhattacharyya) important for sequential decoding, which is not efficient for rates larger R computational cut-off rate Lower bound for = G X o P y X R E () ma E, X ma l g r y E ( R ) R R G c c R E G ( R ) ρ ρ c ρ Rcrit R After maimization the Gallager eponent depends only linearly on the code rate R c C E G ( R c ) for R c C E ( R ) for R C G c c 48 Rc

49 Interpretation Question: What is the maimum code rate R c with E G (R c ) From previous figure the maimum is achieved for = Due to / application of l Hospital X E (, X ) E (, ) y l im l og X ; y y c,ma R ma X ; Y C X For each code rate R c = k/n < C a (n,k) block code eists with a word error probability ne G ( R c ) P w Gallager eponent provides a statement not only regarding the channel, but also about the word error rate P w and the block length n y Y 49

50 Interpretation R E ( R ) G c Observation E G ( R c ) for R c C E ( R ) for R C G c c By increasing n for R c < C the error probability P w can be arbitrarily reduced For R c C the Gallager eponent E G (R c ) approaches zero and n must tend to infinity in order to achieve a small error probability n ( R For = the following relation holds R ) c The closer R c is to R the longer the code word length n has to be chosen in order to achieve the same P w Three regions < R c < R : error probability P w is bounded by n and difference R -R c bound can be computed R < R c < C: error probability P w is bounded by n and error eponent E G (R c ) bound is almost not computable C < R c : error probability P w can not be decreased arbitrarily P w P w ne G c ( R ) ρ Rcrit R C Rc 5

51 Comparison of Channel Capacity and Cut-Off Rate for Binary Symmetric Channel (BSC) R for discrete memoryless channel (DMC) with symmetric binary input ({}={}=.5) R E G () log y y log y y 4 y log y y y C R { }= { }= -P e, { }= { }=P e R log P e P e Comparison shows that R is weaker than capacity term P e 5

52 APPENDIX 5

53 Epectation () Discrete random variable (RV) X taking values in the alphabet ={X,X,X,...} with probability {X=X }={X } Epected value of function f() E X f X X f X Mean value of X: f() = Continuous RV: The epected value of a measurable function f(x) of X, given that X has a probability density function p(), is given by the inner product of p and f: E X f X f p d Mean value of X: f() = E X X X X E E X X p d Variance of X:f() = (-) X X X p d 53

54 Epectation () Linearity: The epected value operator (or epectation operator) E is linear E E X c X c X Y X Y E E E Note that the second result is valid even if X is not statistically independent of Y. Combining the results from previous three equations, we can see that E X Y c E X E Y c 54

55 Rules for Logarithm The logarithm of to base, denoted log, is the unique real number such that Bases : common logarithm : natural logarithm log ln : binary logarithm log ld Change of base Rules ylog log log log log log oduct log log log Power log log Quotient log log log Root log log Eample: ln ln log log log log 55

56 Conveity and Jensens Inequality Conve function over interval, fulfills for every,,and ep i.e. the line segment between,, lies above the graph and The second derivative of conve function is non-negative over the interval Eamples:,, and log, log for Function is concave if is conve Eamples: log, 56

57 Jensen Inequality If is a conve function and is a random variable the inequality holds The inequality states that the conve transformation of a mean is less than or equal to the mean applied after conve transformation Physical version: If a collection of masses are placed on a conve curve at locations,, then the resulting center of mass given by, ) lies above the curve Eample: variables,,with.5 If is a concave function and is a random variable 57

58 Entropy of sink Irrelevance BSC Capacity () log log log Y E Y Y Y Y Y log log bit / s/ Hz YX E log YX X, Y log Y X X Y Y X X Y Y X X Y Y X X Y Y X, log, log, log, log P e log P e P e log P e P e log P e P e log P e P log P P log P P e e e e e Capacity BSC C Y Y X P e P e P e P e P e log log ( ) 58

59 BSC Capacity () BSC: y e with e ~ Bern(P e ) (Bernoulli distribution) Irrelevance YX X EX XX EX E P P log P P log P e e e e e Sink entropy: For ~ Bern(.5) y ~ Bern(.5) Capacity C sup ( X ; Y ) sup ( Y ) ( Y X ) { X } { X } sup ( Y ) ( P ) { X } ( P ) e P P P P log log e e e e e 59

One Lesson of Information Theory

Institut für One Lesson of Information Theory Prof. Dr.-Ing. Volker Kühn Institute of Communications Engineering University of Rostock, Germany Email: volker.kuehn@uni-rostock.de http://www.int.uni-rostock.de/