ABriefReviewof CodingTheory

Transcription

1 ABriefReviewof CodingTheory Pascal O. Vontobel JTG Summer School, IIT Madras, Chennai, India June 16 19, 2014

2 ReliableCommunication Oneofthemainmotivationforstudyingcodingtheoryisthedesireto reliably transmit information over noisy channels. Stream of input symbols Noisy Channel Stream of output symbols

3 DiscreteMemorylessChannels (Part 1) x n, x n 1,..., x 3, x 2, x 1 y n, y n 1,..., y 3, y 2, y 1 DMC A simple class of channel models is the class of discrete memoryless channels(dmcs). A DMC is a statistical channel model that is characterized by

4 DiscreteMemorylessChannels (Part 1) x n, x n 1,..., x 3, x 2, x 1 y n, y n 1,..., y 3, y 2, y 1 DMC A simple class of channel models is the class of discrete memoryless channels(dmcs). A DMC is a statistical channel model that is characterized by a discrete(possibly countably infinite) input alphabet X,

5 DiscreteMemorylessChannels (Part 1) x n, x n 1,..., x 3, x 2, x 1 y n, y n 1,..., y 3, y 2, y 1 DMC A simple class of channel models is the class of discrete memoryless channels(dmcs). A DMC is a statistical channel model that is characterized by a discrete(possibly countably infinite) input alphabet X, a discrete(possibly countably infinite) output alphabet Y,

6 DiscreteMemorylessChannels (Part 2) x n, x n 1,..., x 3, x 2, x 1 y n, y n 1,..., y 3, y 2, y 1 DMC (list continued)

7 DiscreteMemorylessChannels (Part 2) x n, x n 1,..., x 3, x 2, x 1 y n, y n 1,..., y 3, y 2, y 1 DMC (list continued) aconditionalprobabilitymassfunction(pmf)p Yi X i (y i x i )that tellsustheprobabilityofobservingtheoutputsymboly i given thattheinputsymbolx i wassent,

8 DiscreteMemorylessChannels (Part 2) x n, x n 1,..., x 3, x 2, x 1 y n, y n 1,..., y 3, y 2, y 1 DMC (list continued) aconditionalprobabilitymassfunction(pmf)p Yi X i (y i x i )that tellsustheprobabilityofobservingtheoutputsymboly i given thattheinputsymbolx i wassent, the fact that the transmission at different time indices is statisticallyindependent,i.e.,usingx (x 1,...,x n )and y (y 1,...,y n )wehave P Y X (y x) = n P Yi X i (y i x i ). i=1

9 TheBinarySymmetricChannel 0 1 ε 0 ε ε 1 1 ε 1 Letε [0,1]. Asimplemodelis,e.g.,thebinarysymmetricchannel(BSC) withcross-overprobabilityε. ItisaDMC

10 TheBinarySymmetricChannel 0 1 ε 0 ε ε 1 1 ε 1 Letε [0,1]. Asimplemodelis,e.g.,thebinarysymmetricchannel(BSC) withcross-overprobabilityε. ItisaDMC withinputalphabetx = {0, 1},

11 TheBinarySymmetricChannel 0 1 ε 0 ε ε 1 1 ε 1 Letε [0,1]. Asimplemodelis,e.g.,thebinarysymmetricchannel(BSC) withcross-overprobabilityε. ItisaDMC withinputalphabetx = {0, 1}, withoutputalphabety = {0, 1},

12 TheBinarySymmetricChannel 0 1 ε 0 ε ε ε Letε [0,1]. Asimplemodelis,e.g.,thebinarysymmetricchannel(BSC) withcross-overprobabilityε. ItisaDMC withinputalphabetx = {0, 1}, withoutputalphabety = {0, 1}, and with conditional probability mass function 1 ε (y i = x i ) P Yi X i (y i x i ) = ε (y i x i ).

13 TheBinaryErasureChannel δ 1 δ δ δ 0 1 Letδ [0,1]. Anotherpopularmodelisthebinaryerasurechannel(BEC) witherasureprobabilityδ. ItisaDMC

14 TheBinaryErasureChannel δ 1 δ δ δ 0 1 Letδ [0,1]. Anotherpopularmodelisthebinaryerasurechannel(BEC) witherasureprobabilityδ. ItisaDMC withinputalphabetx = {0, 1},

15 TheBinaryErasureChannel δ 1 δ δ δ 0 1 Letδ [0,1]. Anotherpopularmodelisthebinaryerasurechannel(BEC) witherasureprobabilityδ. ItisaDMC withinputalphabetx = {0, 1}, withoutputalphabety = {0,, 1},

16 TheBinaryErasureChannel 0 1 δ δ 0 δ δ Letδ [0,1]. Anotherpopularmodelisthebinaryerasurechannel(BEC) witherasureprobabilityδ. ItisaDMC withinputalphabetx = {0, 1}, withoutputalphabety = {0,, 1}, and with conditional probability mass function 1 δ (y i = x i ) P Yi X i (y i x i ) = δ (y i = ).

17 TheBinary-InputAWGNC Letσ 2 beanon-negativerealnumber.yetanotherpopularmodel (which is strictly speaking not a DMC, though) is the binary-input additive white Gaussian noise channel(awgnc). It is a memoryless channel model

18 TheBinary-InputAWGNC Letσ 2 beanon-negativerealnumber.yetanotherpopularmodel (which is strictly speaking not a DMC, though) is the binary-input additive white Gaussian noise channel(awgnc). It is a memoryless channel model withdiscreteinputalphabetx = {0, 1},

19 TheBinary-InputAWGNC Letσ 2 beanon-negativerealnumber.yetanotherpopularmodel (which is strictly speaking not a DMC, though) is the binary-input additive white Gaussian noise channel(awgnc). It is a memoryless channel model withdiscreteinputalphabetx = {0, 1}, with continuous output alphabet Y = R,

20 TheBinary-InputAWGNC Letσ 2 beanon-negativerealnumber.yetanotherpopularmodel (which is strictly speaking not a DMC, though) is the binary-input additive white Gaussian noise channel(awgnc). It is a memoryless channel model withdiscreteinputalphabetx = {0, 1}, with continuous output alphabet Y = R, and with conditional probability density function p Yi X i (y i x i ) = 1 exp ( (y ) i x i ) 2 2πσ 2σ 2, where x i 1 2x i +1 (x i = 0) 1 (x i = 1).

21 UncodedTransmission x n, x n 1,..., x 3, x 2, x 1 y n, y n 1,..., y 3, y 2, y 1 DMC Consider a BSC with cross-over probability ε [0, 1/2]. Assume that we use uncoded transmission, i.e., we directly send the information bits over the BSC.

22 UncodedTransmission x n, x n 1,..., x 3, x 2, x 1 y n, y n 1,..., y 3, y 2, y 1 DMC Consider a BSC with cross-over probability ε [0, 1/2]. Assume that we use uncoded transmission, i.e., we directly send the information bits over the BSC. Ourbestdecisionaboutx i willbe ˆx i y i.

23 UncodedTransmission x n, x n 1,..., x 3, x 2, x 1 y n, y n 1,..., y 3, y 2, y 1 DMC Consider a BSC with cross-over probability ε [0, 1/2]. Assume that we use uncoded transmission, i.e., we directly send the information bits over the BSC. Ourbestdecisionaboutx i willbe ˆx i y i. Itiseasilyseenthattheerrorprobabilityis ( ) Pr ˆXi X i = ε.

24 ABetterApproach (Part 1) {u t } t {u tk+1,..., u tk+k } t {(x tn+1,..., x tn+n )} t Parser Encoder DMC {(û tk+1,..., û tk+k )} t {(ˆx tn+1,..., ˆx tn+n )} t Decoder {(y tn+1,..., y tn+n )} t

25 ABetterApproach (Part 1) {u t } t {u tk+1,..., u tk+k } t {(x tn+1,..., x tn+n )} t Parser Encoder DMC {(û tk+1,..., û tk+k )} t {(ˆx tn+1,..., ˆx tn+n )} t Decoder {(y tn+1,..., y tn+n )} t Firstly, we parse the string of information symbols into blocks of length k.

26 ABetterApproach (Part 1) {u t } t {u tk+1,..., u tk+k } t {(x tn+1,..., x tn+n )} t Parser Encoder DMC {(û tk+1,..., û tk+k )} t {(ˆx tn+1,..., ˆx tn+n )} t Decoder {(y tn+1,..., y tn+n )} t Firstly, we parse the string of information symbols into blocks of length k. Secondly, instead of sending the components of the informationword (u tk+1,...,u tk+k ) overthechannel,wemap(encode)theinformationwordtoa codeword ( ) x tn+1,...,x tn+n, whose components are then sent over the channel.

27 ABetterApproach (Part 2) {u t } t {u tk+1,..., u tk+k } t {(x tn+1,..., x tn+n )} t Parser Encoder DMC {(û tk+1,..., û tk+k )} t {(ˆx tn+1,..., ˆx tn+n )} t Decoder {(y tn+1,..., y tn+n )} t

28 ABetterApproach (Part 2) {u t } t {u tk+1,..., u tk+k } t {(x tn+1,..., x tn+n )} t Parser Encoder DMC {(û tk+1,..., û tk+k )} t {(ˆx tn+1,..., ˆx tn+n )} t Decoder {(y tn+1,..., y tn+n )} t Basedonthe observed channel output ( ytn+1,...,y tn+n )

29 ABetterApproach (Part 2) {u t } t {u tk+1,..., u tk+k } t {(x tn+1,..., x tn+n )} t Parser Encoder DMC {(û tk+1,..., û tk+k )} t {(ˆx tn+1,..., ˆx tn+n )} t Decoder {(y tn+1,..., y tn+n )} t Basedonthe observed channel output ( ytn+1,...,y tn+n ) wemakeadecision (û tk+1,...,û tk+k ) abouttheinformationvector (u tk+1,...,u tk+k ),

30 ABetterApproach (Part 2) {u t } t {u tk+1,..., u tk+k } t {(x tn+1,..., x tn+n )} t Parser Encoder DMC {(û tk+1,..., û tk+k )} t {(ˆx tn+1,..., ˆx tn+n )} t Decoder {(y tn+1,..., y tn+n )} t Basedonthe observed channel output ( ytn+1,...,y tn+n ) wemakeadecision (û tk+1,...,û tk+k ) abouttheinformationvector (u tk+1,...,u tk+k ), or a decision (ˆxtn+1,...,ˆx tn+n ) aboutthecodeword ( x tn+1,...,x tn+n ).

31 ABetterApproach (Part 3) Considerthefollowingen-/de-codingschemewithU = X = {0,1},k = 1, andn = 5thatisusedfordatatransmissionoveraBSCwithcross-over probabilityε [0, 1/2]. (Withoutlossofgenerality,wecanfocusont = 0.) u 1 (u 1 ) (x 1,..., x 5 ) Parser Encoder BSC (û 1 ) (ˆx 1,..., ˆx 5 ) Decoder (y 1,..., y 5 )

32 ABetterApproach (Part 3) Considerthefollowingen-/de-codingschemewithU = X = {0,1},k = 1, andn = 5thatisusedfordatatransmissionoveraBSCwithcross-over probabilityε [0, 1/2]. (Withoutlossofgenerality,wecanfocusont = 0.) u 1 (u 1 ) (x 1,..., x 5 ) Parser Encoder BSC (û 1 ) (ˆx 1,..., ˆx 5 ) Decoder (y 1,..., y 5 ) If(u 1 ) = (0)thenwesendthecodewordx = (0, 0, 0, 0, 0).

33 ABetterApproach (Part 3) Considerthefollowingen-/de-codingschemewithU = X = {0,1},k = 1, andn = 5thatisusedfordatatransmissionoveraBSCwithcross-over probabilityε [0, 1/2]. (Withoutlossofgenerality,wecanfocusont = 0.) u 1 (u 1 ) (x 1,..., x 5 ) Parser Encoder BSC (û 1 ) (ˆx 1,..., ˆx 5 ) Decoder (y 1,..., y 5 ) If(u 1 ) = (0)thenwesendthecodewordx = (0, 0, 0, 0, 0). If(u 1 ) = (1)thenwesendthecodewordx = (1, 1, 1, 1, 1).

34 ABetterApproach (Part 3) Considerthefollowingen-/de-codingschemewithU = X = {0,1},k = 1, andn = 5thatisusedfordatatransmissionoveraBSCwithcross-over probabilityε [0, 1/2]. (Withoutlossofgenerality,wecanfocusont = 0.) u 1 (u 1 ) (x 1,..., x 5 ) Parser Encoder BSC (û 1 ) (ˆx 1,..., ˆx 5 ) Decoder (y 1,..., y 5 ) If(u 1 ) = (0)thenwesendthecodewordx = (0, 0, 0, 0, 0). If(u 1 ) = (1)thenwesendthecodewordx = (1, 1, 1, 1, 1). Weusethedecoder (0) if y contains more zeros than ones (û 1 ) = (1) ifycontainsmoreonesthanzeros.

35 ABetterApproach (Part 4) For obvious reasons, the above coding scheme is called a repetition code.

36 ABetterApproach (Part 4) For obvious reasons, the above coding scheme is called a repetition code. TherateofthecodeisR = k/n = 1/5.

37 ABetterApproach (Part 4) For obvious reasons, the above coding scheme is called a repetition code. TherateofthecodeisR = k/n = 1/5. The error probability is ) ( ) 5 Pr (Û1 U 1 = (1 ε) 2 ε ( ) 5 (1 ε) 1 ε ( ) 5 (1 ε) 0 ε 5, 5 whichforsmallεisclearlysmallerthanintheuncodedcase,butwehave to pay for this improvement by sending more symbols over the channel.

38 ABetterApproach (Part 4) For obvious reasons, the above coding scheme is called a repetition code. TherateofthecodeisR = k/n = 1/5. The error probability is ) ( ) 5 Pr (Û1 U 1 = (1 ε) 2 ε ( ) 5 (1 ε) 1 ε ( ) 5 (1 ε) 0 ε 5, 5 whichforsmallεisclearlysmallerthanintheuncodedcase,butwehave to pay for this improvement by sending more symbols over the channel. Despite this initial success, one has the feeling that one could construct muchbetterrate-1/5codesbytakingkandnlargerwithn = 5k.

39 ABetterApproach (Part 5) Thecode(orcodebook)isthesetofallcodewords: C { x X n thereexistsan u U k s.t. x = Encoder(u) }

40 ABetterApproach (Part 5) Thecode(orcodebook)isthesetofallcodewords: C { x X n thereexistsan u U k s.t. x = Encoder(u) } The dimensionless rate of the code is R k n.

41 ABetterApproach (Part 5) Thecode(orcodebook)isthesetofallcodewords: C { x X n thereexistsan u U k s.t. x = Encoder(u) } The dimensionless rate of the code is R k n. Thedimensionedrateofthecodeis R klog 2 U n [bits per channel use].

42 ABetterApproach (Part 5) Thecode(orcodebook)isthesetofallcodewords: C { x X n thereexistsan u U k s.t. x = Encoder(u) } The dimensionless rate of the code is R k n. Thedimensionedrateofthecodeis R klog 2 U n [bits per channel use]. Notethatif U = 2thenthedimensionlessandthedimensionedrate areequal.inthefollowing,wewillmostlydealwiththecase U = X = 2andsowewillsimplytalkabouttherateR.

43 ABetterApproach (Part 6) An important quantity characterizing a code is the minimum Hamming distance d min (C) min x, x C x x d H (x,x ), whered H (x,x )isthehammingdistancebetweenxandx.

44 ABetterApproach (Part 6) An important quantity characterizing a code is the minimum Hamming distance d min (C) min x, x C x x d H (x,x ), whered H (x,x )isthehammingdistancebetweenxandx. Foralinearblockcodewehave d min (C) = min x C x 0 w H (x), wherew H (x)isthehammingweightofx.

45 InformationTheory (Part 1) {u t } t {u tk+1,..., u tk+k } t {(x tn+1,..., x tn+n )} t Parser Encoder DMC {(û tk+1,..., û tk+k )} t {(ˆx tn+1,..., ˆx tn+n )} t Decoder {(y tn+1,..., y tn+n )} t What does information theory tell us about our setup?

46 InformationTheory (Part 1) {u t } t {u tk+1,..., u tk+k } t {(x tn+1,..., x tn+n )} t Parser Encoder DMC {(û tk+1,..., û tk+k )} t {(ˆx tn+1,..., ˆx tn+n )} t Decoder {(y tn+1,..., y tn+n )} t What does information theory tell us about our setup? = ToeveryDMCwecanassociateanumbercalledthe capacity C [bits per channel use].

47 InformationTheory (Part 2) Channel Coding Theorem Letthe(dimensioned)rateRbesuchthatR < C.

48 InformationTheory (Part 2) Channel Coding Theorem Letthe(dimensioned)rateRbesuchthatR < C. Fixanarbitraryǫ > 0.

49 InformationTheory (Part 2) Channel Coding Theorem Letthe(dimensioned)rateRbesuchthatR < C. Fixanarbitraryǫ > 0. Then there exists a sequence of encoders/decoders with informationwordlengthk l andblocklengthn l with R = k llog 2 ( U ) n l

50 InformationTheory (Part 2) Channel Coding Theorem Letthe(dimensioned)rateRbesuchthatR < C. Fixanarbitraryǫ > 0. Then there exists a sequence of encoders/decoders with informationwordlengthk l andblocklengthn l with R = k llog 2 ( U ) n l such that the block error probability fulfills ) ) Pr((Û1,...,Ûk l (U 1,...,U kl ) < ǫ ask l (andthereforeasn l ).

51 InformationTheory (Part 3) Converse to the Channel Coding Theorem Letthe(dimensioned)rateRbesuchthatR > C.

52 InformationTheory (Part 3) Converse to the Channel Coding Theorem Letthe(dimensioned)rateRbesuchthatR > C. Then for any sequence of encoders/decoders with information wordlengthk l andblocklengthn l with R = k llog 2 ( U ) n l

53 InformationTheory (Part 3) Converse to the Channel Coding Theorem Letthe(dimensioned)rateRbesuchthatR > C. Then for any sequence of encoders/decoders with information wordlengthk l andblocklengthn l with R = k llog 2 ( U ) n l the block error probability Pr((Û1,...,Ûk l ) ) (U 1,...,U kl ) isstrictlyboundedawayfromzeroforanyk l (andthereforealso foranyn l ).

54 InformationTheory (Part 3) Converse to the Channel Coding Theorem Letthe(dimensioned)rateRbesuchthatR > C. Then for any sequence of encoders/decoders with information wordlengthk l andblocklengthn l with R = k llog 2 ( U ) n l the block error probability Pr((Û1,...,Ûk l ) ) (U 1,...,U kl ) isstrictlyboundedawayfromzeroforanyk l (andthereforealso foranyn l ). Formoreprecisestatements,see,e.g.,CoverandThomas[1].

55 InformationTheory (Part 4) Notethatthechannelcodingtheoremisapurelyexistentialresultandisbasedontheuseofso-called random codes, i.e., one can show that the average random code is good enough under maximum likelihood(ml) decoding.

56 InformationTheory (Part 4) Notethatthechannelcodingtheoremisapurelyexistentialresultandisbasedontheuseofso-called random codes, i.e., one can show that the average random code is good enough under maximum likelihood(ml) decoding. Arandomcodecanbeconstructedasfollows: the?-entriesintheencodingtablebelowmustbefilledwith randomlyselectedelementsofx. (Hereshownfor U = {0,1},k = 3,andn = 5). (u 1,u 2,u 3 ) (x 1,x 2,x 3,x 4,x 5 ) (0, 0, 0) (?,?,?,?,?) (0, 0, 1) (?,?,?,?,?) (0, 1, 0) (?,?,?,?,?) (0, 1, 1) (?,?,?,?,?) (1, 0, 0) (?,?,?,?,?) (1, 0, 1) (?,?,?,?,?) (1, 1, 0) (?,?,?,?,?) (1, 1, 1) (?,?,?,?,?)

57 InformationTheory (Part 4) Notethatthechannelcodingtheoremisapurelyexistentialresultandisbasedontheuseofso-called random codes, i.e., one can show that the average random code is good enough under maximum likelihood(ml) decoding. Arandomcodecanbeconstructedasfollows: the?-entriesintheencodingtablebelowmustbefilledwith randomlyselectedelementsofx. (Hereshownfor U = {0,1},k = 3,andn = 5). (u 1,u 2,u 3 ) (x 1,x 2,x 3,x 4,x 5 ) (0, 0, 0) (?,?,?,?,?) (0, 0, 1) (?,?,?,?,?) (0, 1, 0) (?,?,?,?,?) (0, 1, 1) (?,?,?,?,?) (1, 0, 0) (?,?,?,?,?) (1, 0, 1) (?,?,?,?,?) (1, 1, 0) (?,?,?,?,?) (1, 1, 1) (?,?,?,?,?) If one wants to generate a sequence of capacity-achieving(c.a.) codes then the?-entries must be filled with randomly and independently selected elements from X according to the so-called c.a. input distribution. Moreover,kandnmustgoto wherebyr = klog 2 ( U )/n.

58 InformationTheory (Part 5) Forthebinary-inputAWGNC,theBSC,andtheBEC,thismeansthatallentries should be randomly and independently chosen such that there are about the samenumberofzerosandones.

59 InformationTheory (Part 5) Forthebinary-inputAWGNC,theBSC,andtheBEC,thismeansthatallentries should be randomly and independently chosen such that there are about the samenumberofzerosandones. However, encoding has extremely high memory complexity because the whole encoding table has to be stored.

60 InformationTheory (Part 5) Forthebinary-inputAWGNC,theBSC,andtheBEC,thismeansthatallentries should be randomly and independently chosen such that there are about the samenumberofzerosandones. However, encoding has extremely high memory complexity because the whole encoding table has to be stored. Moreover, ML decoding(or even some sub-optimal decoding) of such a code has extremely high memory and computational complexity.

61 InformationTheory (Part 5) Forthebinary-inputAWGNC,theBSC,andtheBEC,thismeansthatallentries should be randomly and independently chosen such that there are about the samenumberofzerosandones. However, encoding has extremely high memory complexity because the whole encoding table has to be stored. Moreover, ML decoding(or even some sub-optimal decoding) of such a code has extremely high memory and computational complexity. Encoding/decoding of such random codes of reasonable length and rate is highly impractical.

62 InformationTheory (Part 5) Forthebinary-inputAWGNC,theBSC,andtheBEC,thismeansthatallentries should be randomly and independently chosen such that there are about the samenumberofzerosandones. However, encoding has extremely high memory complexity because the whole encoding table has to be stored. Moreover, ML decoding(or even some sub-optimal decoding) of such a code has extremely high memory and computational complexity. Encoding/decoding of such random codes of reasonable length and rate is highly impractical. Weneedcodeswithmorestructure!

63 InformationTheory (Part 5) Forthebinary-inputAWGNC,theBSC,andtheBEC,thismeansthatallentries should be randomly and independently chosen such that there are about the samenumberofzerosandones. However, encoding has extremely high memory complexity because the whole encoding table has to be stored. Moreover, ML decoding(or even some sub-optimal decoding) of such a code has extremely high memory and computational complexity. Encoding/decoding of such random codes of reasonable length and rate is highly impractical. Weneedcodeswithmorestructure! Luckily, the channel coding theorem imposes only small constraints on the codes,i.e.,itleavesalotoffreedomindesigninggoodcodes.

64 CodingTheory (Part 1.0) In order to obtain practical encoding and coding schemes, people have restricted themselves to certain classes of codes that have some structure that can be exploited for encoding/decoding.

65 CodingTheory (Part 1.0) In order to obtain practical encoding and coding schemes, people have restricted themselves to certain classes of codes that have some structure that can be exploited for encoding/decoding. Hereweonlydiscussthecase U = X = {0,1}.

66 CodingTheory (Part 1.0) In order to obtain practical encoding and coding schemes, people have restricted themselves to certain classes of codes that have some structure that can be exploited for encoding/decoding. Hereweonlydiscussthecase U = X = {0,1}. Of course, by restricing oneself to certain classes of codes, it can happen that one loses in performance compared to the the best possible coding scheme where no restrictions are imposed on the encoding and decoding complexity.

67 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2.

68 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2. This allows one to use results from linear algebra.

69 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2. This allows one to use results from linear algebra. Encodingcanbecharacterizedbyak nmatrixgoverf 2 : { C = x F n } 2 thereexistsan u F k 2 suchthat x = u G. G is called the generator matrix.

70 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2. This allows one to use results from linear algebra. Encodingcanbecharacterizedbyak nmatrixgoverf 2 : { C = x F n } 2 thereexistsan u F k 2 suchthat x = u G. G is called the generator matrix. ThecodeC isak-dimensionalsubspaceoff n 2. Theparameterkis therefore often called the dimension of the code.

71 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2. This allows one to use results from linear algebra. Encodingcanbecharacterizedbyak nmatrixgoverf 2 : { C = x F n } 2 thereexistsan u F k 2 suchthat x = u G. G is called the generator matrix. ThecodeC isak-dimensionalsubspaceoff n 2. Theparameterkis therefore often called the dimension of the code. Arank-(n k)matrixhofsizem noverf 2 suchthat { C = x F n } 2 x H T = 0. iscalledaparity-checkmatrix.notethatm n k.(itisclearthatfora given code C there are many possible parity-check matrices.)

72 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2 (continued).

73 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2 (continued). Some simplifications can be done in the ML decoder.

74 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2 (continued). Some simplifications can be done in the ML decoder. The all-zero word is always a codeword. For analysis purposes, we can always assume that the all-zero codeword was sent. (For this statement we assumed that the channel is output-symmetric and that the decoder is symmetric.)

75 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2 (continued). Some simplifications can be done in the ML decoder. The all-zero word is always a codeword. For analysis purposes, we can always assume that the all-zero codeword was sent. (For this statement we assumed that the channel is output-symmetric and that the decoder is symmetric.) The resulting codes are called binary linear block codes.

76 CodingTheory (Part 1.1) Restriction: encodingmapislinearoverf 2 (continued). Some simplifications can be done in the ML decoder. The all-zero word is always a codeword. For analysis purposes, we can always assume that the all-zero codeword was sent. (For this statement we assumed that the channel is output-symmetric and that the decoder is symmetric.) The resulting codes are called binary linear block codes. A binary linear code of length n, dimension k, and minimum distance d min iscalledan[n,k]binarylinearcodeoran[n,k,d min ]binarylinear code.

77 CodingTheory (Part 2) Restriction: encodingmapislinearoverf 2 and cyclic shifts of codewords are again codewords.

78 CodingTheory (Part 2) Restriction: encodingmapislinearoverf 2 and cyclic shifts of codewords are again codewords. This allows one to use results from linear algebra and results about polynomials. Fundamental theorem of algebra, discrete Fourier transform.

79 CodingTheory (Part 2) Restriction: encodingmapislinearoverf 2 and cyclic shifts of codewords are again codewords. This allows one to use results from linear algebra and results about polynomials. Fundamental theorem of algebra, discrete Fourier transform. Encoding can be characterized by a monic degree-(n k) polynomial g(x) F 2 [X]: C = c(x) F 2[X] thereexistsan u(x) F 2 [X] s.t. deg(u(x)) < k ands.t. c(x) = u(x) g(x). g(x) is called the generator polynomial.

80 CodingTheory (Part 2) Restriction: encodingmapislinearoverf 2 and cyclic shifts of codewords are again codewords(continued). Thereisamonicdegree-kpolynomialh(X) F 2 [X]suchthat C = c(x) F deg(c(x)) < n 2[X] c(x) h(x) = 0(mod X n 1). h(x) is called the parity-check polynomial.

81 CodingTheory (Part 2) Restriction: encodingmapislinearoverf 2 and cyclic shifts of codewords are again codewords(continued). Thereisamonicdegree-kpolynomialh(X) F 2 [X]suchthat C = c(x) F deg(c(x)) < n 2[X] c(x) h(x) = 0(mod X n 1). h(x) is called the parity-check polynomial. Encoding can be done very efficiently(especially in hardware).

82 CodingTheory (Part 2) Restriction: encodingmapislinearoverf 2 and cyclic shifts of codewords are again codewords(continued). Thereisamonicdegree-kpolynomialh(X) F 2 [X]suchthat C = c(x) F deg(c(x)) < n 2[X] c(x) h(x) = 0(mod X n 1). h(x) is called the parity-check polynomial. Encoding can be done very efficiently(especially in hardware). Theresultingclassofcodesiscalledcyclicblockcodes.

83 CodingTheory (Part 3) Some remarks:

84 CodingTheory (Part 3) Some remarks: Cyclic block codes have traditionally been one of the most popular classes of codes. Reed-Solomon codes, BCH codes, Reed-Muller codes, etc.

85 CodingTheory (Part 3) Some remarks: Cyclic block codes have traditionally been one of the most popular classes of codes. Reed-Solomon codes, BCH codes, Reed-Muller codes, etc. Within the class of linear block codes there are many special classes, e.g., the class of algebraic-geometry codes. (Here one can use the powerful Riemann-Roch Theorem.)

86 CodingTheory (Part 3) Some remarks: Cyclic block codes have traditionally been one of the most popular classes of codes. Reed-Solomon codes, BCH codes, Reed-Muller codes, etc. Within the class of linear block codes there are many special classes, e.g., the class of algebraic-geometry codes. (Here one can use the powerful Riemann-Roch Theorem.) Etc.

87 CodingTheory (Part 3) Some remarks: Cyclic block codes have traditionally been one of the most popular classes of codes. Reed-Solomon codes, BCH codes, Reed-Muller codes, etc. Within the class of linear block codes there are many special classes, e.g., the class of algebraic-geometry codes. (Here one can use the powerful Riemann-Roch Theorem.) Etc. See, e.g., the book by MacWilliams and Sloane[2] that contains many results on traditional coding theory.

88 CodingTheory (Part 4) Modern coding theory is based on codes that have a sparse graphical representation with small state-space sizes.

89 CodingTheory (Part 4) Modern coding theory is based on codes that have a sparse graphical representation with small state-space sizes. For such codes, very efficient, although usually suboptimal, decoding algorithms are known(sum-product algorithm decoding, min-sum algorithm decoding, etc.).

90 CodingTheory (Part 4) Modern coding theory is based on codes that have a sparse graphical representation with small state-space sizes. For such codes, very efficient, although usually suboptimal, decoding algorithms are known(sum-product algorithm decoding, min-sum algorithm decoding, etc.). Designing good codes is about finding graphical representations where these decoding algorithms work well.

91 Traditional vs. Modern CodingandDecoding Code design Decoding Traditional Reed-Solomon codes etc.? Modern? Iterative decoding (Sum-product algorithm, etc.)

92 Traditional vs. Modern CodingandDecoding Code design Decoding Traditional Reed-Solomon codes etc. Berlekamp-Massey decoder etc. Modern Codes on Graphs (LDPC/Turbo codes, etc.) Iterative decoding (Sum-product algorithm, etc.)

93 TheLawofLargeNumbers The channel coding theorem and many other results in information theory rely on the law of large numbers. That is why coding/decoding works better the longer the codes are. However, in many practical applications one wants to limit delays. Giventhis,codesusedinpracticetypicallyhaveblocklengthsofafew hundreds up to a few thousands(and sometimes a few ten thousands).

94 References [1] T.M.CoverandJ.A.Thomas,ElementsofInformationTheory. Wiley Series in Telecommunications, New York: John Wiley& Sons Inc., A Wiley-Interscience Publication. [2] F. J. MacWilliams and N. J. A. Sloane, The Theory of Error-Correcting Codes. New York: North-Holland, [3] J. L. Massey, Applied Digital Information Theory I and II. Lecture Notes, ETH Zurich, Available online under free_docs.en.html.